Metrics App & Protocol: Reviewer Network(s)

MetricsDAO
1

As of now, the MetricsDAO Bounty Ops team asks for Reviewer volunteers and then makes a selection of who will review. We often see 30-50 volunteers but the existing system only allows for 3 reviewers at a time.

When launching a Challenge with the new app, a specific Reviewer network could be selected as the group responsible for Review…

This is made possible by the app + trybadger.com

image
image786×173 22.5 KB

What does this all mean?

We have new design space!

Reviewer networks can begin to organize!

While it makes sense to maintain a generalized Reviewer network, there is also upside in specialized Reviewer groups.

  • How should the Generalized Reviewer network be paid? Hourly? Per Review?
  • Who should be a Generalized Reviewer? What should be the requirements?
  • What should be the process for losing the right to review?
  • What Specialized Reviewer groups should exist in the future? Who should be on them? How should they be paid? How should membership be maintained?

I want to note that due to the permissionless nature of these systems, I expect a future where there are multiple self-identifying Reviewer groups that vie for the opportunity to be selected by those that launch challenges.

Please also contribute your opinion on this post about the new Reviewer scale: Metrics App & Protocol: Reviewer Scoring

More information about multi-network coordination made possible with the future Metrics protocol:

2

One of the networks that I would like to recognize is PineDAO. Pine runs contests for on-chain analysts.

Pine has two networks that I would feel comfortable using as a Reviewer Network if I was launching a Challenge:

Pine Champion

This Badge is for winners of the contests. There are currently 13 holders.

Address: 0x83c923ed97a14271610b8125317a85f469a18662
Token: 0

Pine Citizen

This Badge is for voters in Pine contests. There are currently 23 holders.

Address: 0x83c923ed97a14271610b8125317a85f469a18662
Token: 1

3

I’ve Badged 137 addresses that were holding over 2000 xMETRIC on Jan 26, 2022.

This is another example network that could be utilized.

4

After talking to @Sunslinger, I want to add some more comments:

Yes, the idea of MANY Reviewer networks is far out.

BUT, our primary Reviewer network is staring us in the face.

For the past few months, the Reviewer network has been defined as participants with 2000 xMETRIC.

This means that anyone can become a Reviewer if they accumulate enough xMETRIC through Analytics participation or through other mechanisms. Keep in mind that the Growth team has been looking for ways to distribute xMETRIC, so there are a variety of ways to earn xMETRIC.

I believe that the existing threshold we’ve used to define our Reviewer network has allowed us to get to this point and to handle the volume we’ve seen but I also think that membership to this network needs to two-way.

In this current state, if we have actors who are not acting in a way that is productive, our recourse would be to slash their xMETRIC. The Bounty Ops or Core team could do this. These users could then regain access by earning more xMETRIC. Playing this out further, there may be a need for a governance process to figure out how to handle this. Governance processes take time and in this case do they involve too many stakeholders?

I am suggesting that if someone takes up the mantle of managing a Reviewer network they can further decentralize this organization.

By moving this ability away from the Core team and creating permissionless rails where anyone can define a Reviewer network, there is no need to utilize the MetricsDAO governance process because each of these networks can be governed on their own based on the design set forth by the network manager.

For example, if a member of the MetricsDAO Bounty Ops team who is familiar with the type of work to be reviewed, the reviewers who are active, and the common pitfalls that may occur then they are likely a better network manager than someone like me who has been more focused on different aspects of the organization and product recently.

5
  • How should the Generalized Reviewer network be paid? Hourly? Per Review?

Have bounty ops staff choose three trusted people who commit to the completion of all the bounties that are assigned to them. Each one of these reviewers reviews as many of the bounties as they have time for, and they also coordinate the rest of the reviewers who

  1. Are paid by the bounty and
  2. Are allowed to commit to reviewing a smaller number of bounties, possibly as few as 1. These people get assigned a batch of 1-5 bounties, depending on what the reviewer commits to from their end and on the level of trust, that the reviewers put in them.

The per-bounty pay should be based on a benchmark average amount of time that is determined to be the optimal balance between a fair and helpful review and taking away too much bounty prize money.

As a reviewer and analyst, my number is 10-15 minutes. So let’s say 4 bucks per bounty. That’s my vote on that, and that may not be the right formula, but decide this and then pay accordingly.
It sucks that if you pay by the hour, reviewers are incentivized to pad their hours. I don’t pad my hours, but I am constantly concerned that I am claiming more money than my peers because I am too thorough. Unfortunately, paying by the bounty means that some uncaring schmuck could just randomly score people without really even reading the bounties and that is unfair. This is why you have to feed them a small number of bounties at a time until they become trustworthy. You also have to pay attention to what they are doing.

Who should be a Generalized Reviewer? What should be the requirements?

I believe the bar, to being eligible to be an entry-level or candidate reviewer, should be much lower than the current 2000 xmetric. A huge part of the job of the analyst is to make their analysis understandable. I think it is therefore a good thing to have people who may not be expert analysts, but who can still tell whether an analysis is worthwhile or not. I think the requirements should be that you have successfully completed one bounty, that got paid. I just think that reviewers can be assessed, and that priority should be given to better reviewers and that, some things are better done in-house and farming out the reviewing job to networks seems dubious to me when it is MDAO that is solely responsible for giving a fair grade in a reasonably fast amount of time.

I have another idea that I am kicking around but I think I like it. Given a bounty, which is to be graded, only those who actually submitted that bounty are eligible. They would be required to grade their own bounty, but it would not count toward their score. I think it is a plus nee almost a necessity that a person has worked the bounty so that they know what the queries are and what the filters should be and will notice if figures seem wrong. Just an idea though …

*What should be the process for losing the right to review?

The choice of who reviews should be at the sole discretion of bounty ops. They should rank reviewers and prioritize based on the quality of their work. You just can’t outsource, decentralize or automate this, I don’t think. Slashing Xmetric sounds like something you do to someone who has done something immoral. I would just not rehire them. If they cheated then yeah slash their xmetric. But just curious, surely everyone has thought about the fact that you can send someone a negative badger, just as easily as you can send a positive badger so that might be better than slashing xmetric.

What Specialized Reviewer groups should exist in the future? Who should be on them?

I don’t know from specialized review groups. I do like tiers based on quality and experience. Use badger for this since it is so easy to revoke these and would make it simple to see your list of folks in each group. Amount of metric could be a factor in determining the tier levels. I have another idea which if implemented, could help with quality and consistency of reviewing, which is to have reviewers scored from one to five. This would be open to any metrics dao participants. My idea is you have an amazon reviews type section. Each bounty has a set of reviews. Most of these would be unofficial. Everyone could add on reviews for no pay, but their participation, and the quality of their reviews could enhance the likelihood of them getting to review bounties themselves.

How should they be paid?

In cold hard USDC. Going with my tiered system you would have to decide how much more an hour their work is worth and up the per bounty pay based on that. pay = hours expected to be spent times what you think their hourly wage is worth.

How should membership be maintained?
With Badger of course! Bounty ops determine that a reviewer is worthy of a promotion. They assign that person a new tier level. This person now maybe gets paid more or gets to grade more advanced bounties or something.

I think we need a proper review form that takes each scoring category and breaks it down into a few subcategories. Reviewers would answer the subcategory questions and then score and comment. They need to do this for each category. They also should make one overall comment about the bounty. This requirement, and only giving them a small batch to work with at a time would force them to slow down a bit. The subcategories would raise alarms about the quality of the reviewer if their answers were way off of everyone else’s. If an analyst has a complaint about a review, they could go to the reviewer first with their complaint. The reviewer then has a chance to explain their grade to the analyst. If the reviewer is unavailable or unwilling to respond or if the answer they give is unsatisfactory, they could then submit a ticket for regrading. This would also be another way to screen out bad reviewers.

I know I have a lot of suggested implementations of things, but I actually think it could be rolled out pretty quickly, with some manual bookkeeping required for a while and eventually smoothing out the process. I have zero understanding, at this point, about what the app will and won’t be able to do, so I realize that even if you love my ideas that it might not be workable given already made decisions about the process.

6

Networks

While exploring the potential of the Reviewer Network, I’m also interested in how networks of different participants can develop and expand organically.

The three key pillars of MetricsDAO ecosystem are Sponsors, Analysts and Reviewers. Sponsors launch analytical challenges, the Analysts solve them and the Reviewers verify the accuracy of the solutions.

In my opinion, the system should allow sponsors, analysts, and reviewers to build their profiles and showcase their skills and work. This would help to create a sense of competition within the platform, enabling sponsors to find the most qualified analysts for their challenges, analysts to gain recognition for their work, and reviewers to be rewarded for their accuracy. It would also create a sense of transparency, allowing participans to assess the quality of work being produced by sponsors, analysts and reviewers.

With insightful profiles available for all three participants of the system, I believe the system would organically evolve into self-organizing groups that form their own networks to coordinate. As an example, Protocol X can select qualified analysts and reviewers for an analytical challenge by scanning publicly available profiles.

Having detailed insights into every participant in the ecosystem will lead to organic growth for sponsors, analyst networks and reviewer networks.This will also lead to better rewards for analysts and reviewers who produce high quality work.

Who reviews the reviewers?

Reviewer reviews greatly influence both the rewards earned by Analysts and the quality of submissions received by Sponsors. Therefore, an effective system for training, coaching, and monitoring reviewers is critical.

We should consider running regular training programs for reviewers and also empower Sponsors and Analysts to give feedback on Reviewers. Just as we have Analyst profiles, we should have Reviewers profiles with metrics like the submissions they reviewed, the scores they awarded and the feedback they received from analysts as well as sponsors.

Reviewer Payment

In my opinion, reviewers should be paid according to how complex the analytical challenge is. It takes more time to review a Mega Dashboard than user growth metrics, for example. Therefore, more incentives are needed to review the Mega Dashboard than simple user growth metrics. We could potentially categorize the analytical questions as Easy, Medium, Hard and Advanced and set the payment based on the category. I lean towards a fixed payment system instead of a time-based payment system.

Revoking Reviewer Access

In the world of self-organizing reviewer networks, each network should have the ability to grant access or revoke access.

If it is expected to take a while for self-organizing reviewer networks to evolve - we should flip the question and explore “How many tiers of reviewers do we need and what is the criteria for promoting reviewers from a lower tier to a higher tier?” .

Neither the reviewer nor the DAO will find it easy to navigate the process of revoking access, so we should define “Reviewer Reputation” points and gate tiers using these points (please note that these are different from Analyst reputation points). Also the Reviewer Reputation points should be based on reviews they performed as well as the feedback received from Analysts & Sponsors.

In the unlikely event when we have to slash reputation, we should define the criteria and metrics clearly. For example, a reviewer may loose 50 points for categorizing the analysis as “Incorrect”. Or a reviewer may loose 5 points for failing to identify plagiarism, etc. On the flip side the reviewers who does stellar job in those categories should earn the rewards points.

1 Like
7

Love these responses from @kakamora and @potmo about how to fully build these networks in the future! This is the right type of thinking as each network can be maintained in its own way.

These networks can have their own rules for entrance/exit and operations.

With regard to the primary network which is currently made up by people with 2000+ xMETRIC, I see over 230 eligible participcants.

image

The largest bucket making up that 230+ metric is the bucket of users with 2000-2999 xMETRIC.

Would the Reviewer network improve if the threshold was raised to 3000 xMETRIC?

This would create a group of 84 eligible reviewers.

I believe that @ltirrell and @kakamora have also looked at data about how different Reviewer spend differing amounts of time on Reviews. That data may also be helpful in identifying a network. In any case, taking a data based approach to the general network makes sense while externally managed networks may have their own subjective or objective criteria.

8

Can you explain the advantages of reviewer networks over the current system? All I can see is a reduction of quality control and consistency. I think it is reasonable to have a flexible rubric and even to have the sponsors provide their own grading rubric, but I still just don’t get how this is a good thing. Not trying to be contrarian here, I just don’t get it.

9

Can you explain the advantages of reviewer networks over the current system?

Totally down to take a shot at it, but before I do that I want to talk about the “current system” a bit.

In my mind, the current system is:

  • Gain entry to ‘MDAO Reviewers’ from earning 2k xMETRIC
  • Reviews become available, ‘MDAO Reviewers’ raise their hand
  • 3 are activated from ‘MDAO Reviewers’

This system kind of has 3 gates, get 2k xMETRIC to become a potential reviewer, raise your hand, then get selected as one of the three.

The first gate isn’t very tough to get through and may not be a good indication of someones ability to Review. It gives us some information about the participant - they have been recognized by MetricsDAO at some point in the past for being a good analyst already or for earning 2k xMETRIC through analytics submissions. While it may indicate their ability as an Analyst, Reviewing can be different.

The second gate is raising their hand, this is a good indication of the participants availability to take on work. It also can indicate how many of the participants are willing to take on work. For example, we often see 20-40 volunteers when Reviews are available so we know 10-20% of ‘MDAO Reviewers’ are available and willing to do Reviews on a pretty consistent basis.

The third gate falls to the Bounty Bunch and I am not sure what their criteria for selection is or what randomization methods they might employ.

This group of individuals with over 2k xMETRIC are part of the network whether or not they know it. They have access to the second gate and can indicate their availability to make it through the third gate.

This system isn’t great. It works! But we’re taking on more volume!

If we had distinct networks, then the Bounty Bunch may have more levers to manage budget or to tap networks that they know deliver consistent results. Imagine if there was a network of 10 Reviewers who were willing to take on reviews at a rate of $5 per review

Competing networks also decentralizes the responsibility away from the Bounty Bunch and can give others the autonomy to become the Don of their own Reviewer Network.

Going to point back to my earlier message:

Yes, the idea of MANY Reviewer networks is far out.
BUT, our primary Reviewer network is staring us in the face.

For the past few months, the Reviewer network has been defined as participants with 2000 xMETRIC. This means that anyone can become a Reviewer if they accumulate enough xMETRIC through Analytics participation or through other mechanisms. Keep in mind that the Growth team has been looking for ways to distribute xMETRIC, so there are a variety of ways to earn xMETRIC.

If we all think the existing system is working fine then I am good to take a step back on this push for a bit and then to re-assess.