mrkeyoor.com_
Tue 22 Sept 06:44 UTC
AI Toolsevaluationupdated 22 Sept 2026

google-research review

This subdirectory contains experiment code for a NeurIPS 2022 paper about learning a yes-or-no classifier when training data arrives in groups with only group-level label proportions. It generates synthetic bags of 3 or 4 points, solves an optimization problem, compares the learned classifier with a random threshold, and writes tables for the paper.

Verdict

Our run installed 35 packages in 7 seconds with 0 known vulnerabilities, but no test target existed to check the 6 experiment scripts. Use this code to inspect or reproduce the NeurIPS 2022 method only if you can repair and independently validate the four-item small-margin path. Build a separate implementation for real data or production work.

We ran it

Lab card: what happened when we ran google-researchScreenshot of google-research (research.google)
Install✓ · 7s35 packages · 37 MB
Build✓ · 0s
Testsn/ano test script
Known vulns0(pip-audit)
Repo23143 files~2,481,843 lines of source · 1283 MB · 0 CI workflows

Answers from our run

Does google-research build from source?

Dependencies installed in 7 seconds (35 packages), and the build succeeded in 0s. We cloned commit e1e8d86 into a clean Debian container with 3 CPUs and no project-specific setup.

Does google-research have tests you can run?

Not through a standard command: the project exposes no test script or target that our harness could run.

Does google-research have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use google-research?

Teams seeking an installable LLP library or service: the subdirectory has 6 standalone experiment scripts, no package API, and no data-ingestion interface.

What are the alternatives to google-research?

Covariate Shifted LLP, LLP-VAT. Our run installed 35 packages in 7 seconds with 0 known vulnerabilities, but no test target existed to check the 6 experiment scripts.

Setup3/57-second install, but the small-margin command is wrong
Docs2/5A 19-line README omits inputs, runtime, and known limits
Community2/5Active monorepo, no issue activity tied to this 2022 artifact
Maturity1/5No tests or CI; the 4-item small-margin path needs repair

Who it’s for

Machine-learning researchers reproducing the NeurIPS 2022 experiments on linear thresholds and label proportions.
Theorists who want runnable CVXPY formulations for synthetic bags of size 3 or 4.
Engineers prepared to extract the method, repair the four-item path, and write their own data-loading and validation layer.

Who it’s NOT for

Teams seeking an installable LLP library or service: the subdirectory has 6 standalone experiment scripts, no package API, and no data-ingestion interface.
Researchers who need to trust the four-item small-margin results unchanged: that script assigns bag["x4"] to x3, never reloads x4 inside the scoring loop, and samples test points with random.randrange(3) from a four-item list.
Readers expecting the README commands to run as written: both small-margin filenames contain LLP-LTF, while the documented command uses LLP_LTF.
Teams that require automated regression evidence: our scan found 0 CI workflows, no tests directory, and no test target to run.

Setup reality

Our sandbox install succeeded in 7 seconds, adding 35 packages and 37 MB. The recorded build completed in 0 seconds. The full checkout contained 23,143 files, about 2,481,843 source lines, and occupied 1,283 MB. There was no test script or target, so tests were skipped; pip-audit found 0 known vulnerabilities.

The experiment needs no credentials or external service. Its requirements file targets Linux and pins Python 3.9.6, CVXPY 1.1.13, NumPy 1.21.1, pandas 1.3.1, and SCS 2.1.4. Parameters such as dimensions, bag counts, runs, and rounding attempts are constants inside the scripts.

The README's small-margin command does not match either checked-in filename. There is no Dockerfile, tests directory, or CI workflow, and the four-item small-margin scoring loop has two source-level indexing problems. Treat setup as the start of a code audit, not proof that the published experiment path is correct.

The 8-file subproject reproduces one NeurIPS 2022 result

Buried inside Google's 23,143-file research monorepo is a much smaller artifact: 8 files devoted to one paper. The problem is learning a linear yes-or-no classifier without seeing each training label. Instead, examples arrive in bags and the learner sees only the proportion of positive labels in each bag. The NeurIPS 2022 paper gives approximation results for this setting, including a semidefinite-programming approach for bags of size 3 and a weaker guarantee for sizes of 4 or more.

The subproject turns part of that theory into 4 experiment scripts and 2 result-processing scripts. Each experiment creates synthetic vectors, builds a CVXPY optimization problem, rounds its matrix solution into a classifier, and compares that classifier with a random threshold. The defaults try dimensions 10 and 40, bag counts 50 and 100, 25 runs, and 5 Gaussian rounding attempts. Two final scripts summarize CSV output and write LaTeX tables. That is useful reproduction material, provided you want the experiment the author encoded.

The 4 experiment scripts have fixed synthetic inputs

There is no library surface to call from an application. The scripts execute their work at module level, with dimensions, bag counts, random seeds, run counts, and solver selection written directly into the source. Changing the study means editing those constants or refactoring the files. No command-line arguments accept a dataset, no documented schema describes bags from disk, and no trained model interface is exposed. The output is a pair of CSV files per bag size, followed by summarized LaTeX tables.

The environment is equally specific. requirements.txt targets Linux, Python 3.9.6, CVXPY 1.1.13, NumPy 1.21.1, pandas 1.3.1, SciPy 1.7.0, and SCS 2.1.4. No account, API key, hosted service, or database is involved. That makes the method inspectable offline. It also leaves environment modernization to the reader, because the README supplies a Conda instruction and a three-command sequence without explaining solver behavior, expected numerical output, or how long the experiments should take.

What happened when we ran it

Our sandbox installed 35 packages in 7 seconds and added 37 MB on disk. The recorded build completed successfully in 0 seconds. We ran commit e1e8d86 in a fresh Debian container with 3 CPUs and 8 GB of RAM, using Python 3.12. The entire checkout was 1,283 MB, with 23,143 files and roughly 2,481,843 lines of source, which supports the root README's advice to download only the subdirectory you need.

There was no test script or target, so our harness skipped tests rather than reporting a pass. The repository scan also found 0 CI workflow files, no Dockerfile, and no tests directory. Pip-audit reported 0 known vulnerabilities among the installed packages. Those results say the dependency set installed cleanly in our container. They do not verify the numerical experiments, compare generated tables with the paper, or show that every script reaches its expected CSV output.

A 7-second install can create false confidence here. The build took 0 seconds because there is no compiled application to assemble, while the substantive work lives in scripts that solve repeated optimization problems. Our measurement did not produce a benchmark or a validated research result, so we will not invent either. Before relying on the method, record solver status, preserve raw CSV files, compare the summaries with the paper, and add checks around bag construction and evaluation.

The 19-line README contains a broken command

For each bag size, the README tells you to run small_margin_q-sized_LLP_LTF.py. Both checked-in small-margin files use a hyphen in LLP-LTF instead, so the documented filename does not exist. The mismatch is easy to repair once spotted, yet it matters in a 19-line setup guide with no troubleshooting section. A reader following the only command example receives no warning that the name differs. The large-margin and processing filenames do match their documented patterns.

The four-item small-margin script has a more serious problem. Inside its scoring loop, it loads bag["x3"] and immediately overwrites that variable with bag["x4"]; it never assigns the current bag's fourth point to x4. Earlier, its four-item test list is sampled with random.randrange(3), which can select only indices 0 through 2. These are source-level correctness defects, not failures observed by the lab harness. They still make that path unsafe to quote without a fix, a rerun, and independent result checks.

The monorepo is active, while this artifact has stood still since 2022

Google's repository was pushed on September 21, 2026, and GitHub listed 38,812 stars plus 1,995 open issues and pull requests when fetched. That activity belongs to a monorepo spanning thousands of projects. The label-proportions subdirectory entered the repository on October 11, 2022. Later path commits in 2023, 2024, 2025, and 2026 changed copyright years rather than experiment logic, and GitHub issue search found no report naming this subdirectory or its small-margin filename.

Apache-2.0 licensing leaves room to reuse the method, but the maintenance record says to take ownership of any fork. For a paper reader, the 6 scripts are short enough to trace and the fixed seeds make their intended experiment easier to follow. For an applied team, Covariate Shifted LLP has dataset preprocessing and training machinery, while LLP-VAT addresses neural image classification. Choose this Google Research artifact when the 2022 linear-threshold result itself is the reason you are here, then verify every output you plan to cite.

Alternatives

ProjectWhat it isPick it when
Covariate Shifted LLPA 2025 research implementation with dataset preprocessing, baselines, training, and evaluation for shifted instance distributions.pick this instead when your LLP study uses real datasets or distribution shift and you can accept an archived research repository.
LLP-VATA PyTorch implementation of consistency-regularized LLP experiments on SVHN, CIFAR-10, and CIFAR-100.pick this instead when image classification and neural consistency training are closer to your problem than linear-threshold theory.

What people are saying

  1. [velocity-scout] google-research/google-research
  2. [github-trending] google-research/timesfm

Sources

  1. Google Research repository
  2. Algorithms and Hardness project README at the measured commit
  3. Pinned experiment requirements
  4. Four-item small-margin experiment source
  5. NeurIPS 2022 paper and abstract
  6. Project path commit history

More ai tools reviews

A2A · awesome-artificial-intelligence · ncnn · OpenCreator · editor · autoclip · the whole board →