标签：#learning

ACL ARR May 2026 reviews are due on July 2. I do not see any reviewer assignement as of today. Will the review period be just 2 weeks in that case? Anyone got papers assigned for reviewing? submitted by /u/Impossible-Garden612 [link] [留言]

After debugging 20+ broken RAG systems, I've identified the 6 decisions that determine whether yours works. Here's how to get each one right. The RAG Developer's Trap Every RAG developer falls into the same trap: you build the basic pipeline, it sort of works, and then you spend weeks tweaking prompt templates — while the real problem sits untouched in your indexing pipeline. The 80/20 rule: 80% of RAG problems come from indexing, not generation. But 80% of debugging effort goes into generation. Let's fix that. Decision 1: Embedding Model — The Single Biggest Lever The mistake: Using all-MiniLM-L6-v2 for Chinese documents because it's the default in every tutorial. Why it's wrong: It's English-trained. Drop it on Chinese text and it loses 30-50% of semantic fidelity. Language Use This Chinese BAAI/bge-large-zh-v1.5 (1024-dim) Chinese + English BAAI/bge-m3 (multilingual + sparse) English text-embedding-3-large Code jina-embeddings-v3 or voyage-code-3 Non-negotiable: Indexing model and query model must be byte-for-byte identical. Switch models = rebuild entire index. Impact: +15-40% Recall@10 for Chinese RAG. Decision 2: Chunk Size — Not a Magic Number Physics: Too small (< 100 tokens) = semantic fragmentation. Too large (> 1000 tokens) = noise injection. Document Type Sweet Spot Overlap FAQ / Short-form 128-256 20 Technical docs 512 50 Long-form articles 768-1024 100 Code Function boundaries 0 The method matters more than the size. Use recursive splitting, not fixed-length: from langchain.text_splitter import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter ( chunk_size = 512 , chunk_overlap = 50 , separators = [ " \n\n " , " \n " , " . " , " " , "" ] ) Impact: +5-15% Recall@10. Decision 3: Index Type — HNSW vs IVF Scale Use Why < 1M vectors HNSW Recall > 0.95 1-5M, RAM tight IVF + PQ 75% memory savings > 5M IVF + PQ + Sharding Horizontal scale Key nuance: HNSW has high insertion cost. Streaming docs → IVF may be better even at small scale. Im

Did anyone else submit to ACM ICMI 2026? The reviews were recently released, and this is my first time submitting to ICMI, so I'm not very familiar with the acceptance patterns. I submitted a long paper and received the following overall ratings: 4 (Probably Accept), 3 (Borderline), 4 (Probably Accept) The reviewer with the highest stated expertise recommended acceptance, while the borderline reviewer had some concerns about soundness but still considered it a nice contribution. For those who have submitted to or reviewed for ICMI before, how would you interpret these scores? Is a 4/3/4 generally considered competitive after rebuttal, or is it still a long shot? Would appreciate any insights from past authors or reviewers. submitted by /u/kanishq95 [link] [留言]

I built a distributed compute grid where your idle laptop runs ML jobs — the orchestrator behind it The pitch: a single FastAPI hub takes compute jobs from ML researchers, and a fleet of home PCs and gaming rigs (RTX 4090s, M2 MacBooks, anything with a GPU and a Python interpreter) polls in, picks up work, and ships results back. A 20% platform fee funds the hub. An interactive dashboard shows the mesh in real time. I have been living inside this codebase for a few weeks. This post is about the part that actually determines whether the thing works or does not — the orchestrator . No frontend, no marketing — just the brain. Live dashboard: man44.zo.space/compute-pool Repo: github.com/AmSach/ComputePool-Grid The problem with "dumb" schedulers The first version of ComputeOrchestrator had a one-line bug that took down a 12-node stress test. Two jobs hit the hub at the same millisecond. Both saw the same node as idle . Both wrote busy to the same row. One node ended up double-allocated, the other starved, and the test logs looked like a hostage negotiation. The fix had to be three things at once: A scoring function that picks the right node, not just the first idle one. An async lock so concurrent submissions cannot race on a single node. A heartbeat monitor that reclaims nodes that ghosted. Here is what it looks like now. The scoring algorithm def _calculate_score ( self , capacity : Dict [ str , Any ], requirements : Dict [ str , Any ]) -> float : """ Heuristic for node-task matching. """ score = 0.0 if capacity . get ( " gpu_vram " , 0 ) >= requirements . get ( " min_vram " , 0 ): score += 10.0 if capacity . get ( " cpu_cores " , 0 ) >= requirements . get ( " min_cores " , 0 ): score += 5.0 return score The weights are deliberately lopsided. A node that satisfies a job's VRAM requirement gets a 2x bonus over a node that just barely has enough cores. The intuition: GPU work is the long pole. If you cannot fit the model in VRAM, nothing else matters, no matter how many

Hi everyone, I’m close to completing my degree in Psychology, and I’m also a Systems Engineering student. is like, roughly comparable to Software Engineering / Computer Science outside Latin America. Although I study engineering, I’m still at an early stage with machine learning, LLMs, AI safety, and related technical topics. My research project is mainly psychology-oriented, but I’d really appreciate recommendations or warnings from a software/technical perspective. I’m working on a project about how AI systems respond to prompts involving psychological distress at different levels of intensity. I’m currently considering ChatGPT, Gemini, Wysa, and Replika, and I’m interested in comparing general-purpose LLMs, mental-health-oriented chatbots, and AI companions. Some aspects I’m thinking about are: How each system handles mental health, self-harm, crisis situations, and psychological/medical advice. whether responses change as the prompt becomes more intense, for example when a normal generated response is replaced by a safety protocol, moderation layer, or crisis-resource response. whether systems respond differently to declarative prompts versus question-based prompts, such as “I feel emotionally overwhelmed” vs. “What should someone do if they feels emotionally overwhelmed?” whether responses differ when distress is explicit, indirect, ambiguous, hypothetical, or written in third person. whether the system provides empathy, psychoeducation, referrals, crisis resources, refusal, redirection, or a combination of these. how to account for technical changes over time, such as model versions, neural network weights, safety layers, moderation classifiers, system prompts, memory/retrieval features, and product-level configurations. whether it is methodologically valid to compare systems with very different technical architectures. I’m not trying to evaluate these systems as therapists or test clinical effectiveness with real patients. The focus is on how they respond lin

Submitted a short theoretical paper to TMLR and got desk-rejected with "does not meet our editorial standards or allow us to assess claims and evidence" and "not a suitable venue for this work." Is this a common outcome for first submissions? Curious what typically drives this kind of rejection, scope mismatch, insufficient experiments, or something else. Not looking to appeal, just trying to understand the bar so I don't waste time on the wrong venue next time. Anyone else gotten this and figured out what the actual issue was? submitted by /u/observer678 [link] [留言]

Surprised there's no real tooling for this given how much research exists on continual learning. Built pyrecall to fill the gap. Snapshots skill scores before/after fine-tuning, flags regressions, rolls back LoRA adapters by name. Fully local, no external APIs. v0.1.0, MIT, pip install pyrecall Curious if anyone has thoughts on the benchmark design that's the part I'm least confident about. https://github.com/Arths17/Pyrecall submitted by /u/Level_Frosting_7950 [link] [留言]

Every hand-tracking demo shows you 21 dots. The interesting part is what nobody shows: turning dots into numbers someone can act on. Dots are a capability, not a product Run any modern hand-tracking model and you get 21 beautifully stable landmarks per hand at 30 FPS. Impressive — and by itself, worthless. No client has ever paid for dots. They pay for measurements : is this clearance compliant, is this part aligned, did this patient's range of motion improve. I learned this on utility infrastructure work, where the deliverable was never "we detected the wire" — it was the attachment height of that wire, and whether it violates clearance rules . Keypoints were step one of three. The demo: live metrics, not just a skeleton My portfolio's keypoint demo derives three measurements per hand, every frame: const wrist = lm [ 0 ]; const palm = distance ( wrist , lm [ 9 ]); // scale reference const pinch = distance ( lm [ 4 ], lm [ 8 ]) / palm ; // thumb tip ↔ index tip The crucial line is the scale reference . Pixel distances are meaningless — they change as you move toward the camera. Dividing by palm length (wrist to middle knuckle) gives a relative measurement that's stable under distance, and multiplying by the average adult palm length (~8.5 cm) converts it into an approximate real-world gap — the demo shows "≈ 3.2 cm" floating on the pinch line. In infrastructure work the same role is played by a known object dimension — a standard crossarm, a pole class height. Every measurement-from-pixels system needs its ruler. Finger counting is a geometric test (is each fingertip farther from the wrist than its middle joint?), and "hand openness" averages fingertip extension — three lines of geometry each, but they convert a model output into a readout a human understands instantly. Honest layering The landmarks come from MediaPipe's pretrained pipeline (palm detector → landmark regressor → gesture classifier, float16, WASM + GPU delegate) — Google's models, credited on the page

Training a network from scratch in raw NumPy, quantizing it to int8, and running it as ~80 lines of dependency-free JavaScript — with a parity test proving the browser matches Python to 1e-6. Why bother? MNIST is a solved problem Digit recognition is the "hello world" of ML — that's exactly why I used it. The model isn't the point. The point is everything around the model, which happens to be the part that matters in production work too: training without a framework, compressing for deployment, running inference in a constrained environment, and proving the deployed system matches the trained one. Training: just NumPy and math The network is a 784→128→64→10 MLP — hand-written forward pass, backpropagation, and Adam optimizer. No autograd, no framework: # backward pass, by hand dz3 = ( probs - y_batch ) / batch_size grads_w [ 2 ] = a2 . T @ dz3 da2 = dz3 @ weights [ 2 ]. T dz2 = da2 * ( z2 > 0 ) # ReLU mask grads_w [ 1 ] = a1 . T @ dz2 ... One trick that matters for a drawing demo specifically: shift augmentation . MNIST digits are centered; humans draw wherever they like. Training on randomly translated copies makes the model tolerant of sloppy placement. Combined with MNIST-style preprocessing at inference (crop to bounding box, scale into a 20×20 box, center by center-of-mass), real-world doodles classify reliably. Final test accuracy: 98.2% . Compression: int8 in 15 lines A float32 weight file would be ~430 KB. Symmetric int8 quantization cuts it ~4×: scale = np . abs ( w ). max () / 127.0 q = np . clip ( np . round ( w / scale ), - 127 , 127 ). astype ( np . int8 ) One scale factor per layer, weights stored as base64 in JSON: 145 KB total , and quantized test accuracy is identical to float — 98.2%. Inference: ~80 lines of plain JavaScript In the browser, the weights are dequantized once on load, and inference is three matrix-vector products with ReLU and a softmax. ~109K multiply-adds — about a microsecond-scale problem for any modern device. No TensorFlow.js (t

Camouflage has always been graded by human eyes. But the thing hunting for you in 2026 is increasingly a detection model — so test against that. The premise Surveillance is automated now: drones, trail cameras, perimeter systems — most of what "sees" runs an object-detection network. Which makes traditional camouflage evaluation (a person squinting at a photo) the wrong test. The right test is adversarial: run the actual detector against your concealment and measure what it finds. That's the whole demo: upload a photo, and an object-detection model hunts for people in it — at four simulated distances — producing a detection-range profile and a stealth score. Simulating distance with pixels You can't move the camera after the photo is taken, but you can simulate the dominant factor in long-range detection: pixels on target . A person at 50 m simply occupies far fewer pixels than at 5 m. So each analysis run downscales the image progressively and re-runs detection: const DISTANCE_LEVELS = [ { label : ' Close (~5m) ' , scale : 1 }, { label : ' Mid (~15m) ' , scale : 0.45 }, { label : ' Far (~30m) ' , scale : 0.22 }, { label : ' Very far (~50m) ' , scale : 0.12 }, ]; for ( const level of DISTANCE_LEVELS ) { const scaled = drawScaled ( image , level . scale ); const detections = await model . detect ( scaled , 10 , 0.15 ); // best 'person' confidence at this simulated range } The output reads like a range card: detected at 5 m with 96% confidence, 41% at 15 m, invisible beyond 30 m. A stealth score aggregates it: how poorly did the adversary see you, averaged across ranges? Honest about the model The detector is COCO-SSD (a pretrained MobileNet-based model from the TensorFlow.js team) running entirely on-device — I didn't train it, and the demo says so on the page. The contribution here is the evaluation framework : using detectors as adversaries, simulating range, and turning subjective "good camo" into a measurable profile. The full version of this concept goes further

Hello everyone, tomorrow I have a meeting with my dissertation supervisor and I wanted to have a dissertation proposal ready. Initially, I moved forward with the following proposal: "Interpreting the Routing Dynamics of Capsule Networks for Explainable AI." My first approach to this topic was to study the paper "Transforming autoencoders," which is the first paper about capsule networks. Next, I did a search on the state of the art of transforming autoencoders and only found 2 papers since 2011. I think I should take advantage of the work I have developed so far on transforming autoencoders and write a dissertation about them. If anyone could take a look at the readme and tell me what they think, I would appreciate it. What do you think? I should suggest another topic involving transforming autoencoders. There isn't much scientific research on them. The professor is approachable, and if I present a good new topic, he'll let me change it! submitted by /u/Future-Persimmon5393 [link] [留言]

After 11 years building e-learning software at a major enterprise, here’s what surprised me when we started shipping GenAI features to real users. The Starting Point Nobody Talks About Most GenAI integration blog posts start with a clean slate — a greenfield app, a fresh codebase, a blank canvas. Real life is messier. When we began integrating generative AI into our e-learning authoring tool, we weren’t starting from scratch. We were dealing with a mature enterprise product — millions of users, legacy architecture decisions made years ago, compliance requirements from Fortune 500 customers, and LMS interoperability standards (SCORM, xAPI) that were designed long before anyone imagined AI-generated content. The challenge wasn’t “how do we call an AI API.” It was “how do we ship AI features into a product that thousands of instructional designers depend on daily, without breaking their workflows or their trust.” Here’s what I learned. Lesson 1: The AI Feature Your Users Want Is Not the One You Think When we first scoped out AI integration, the engineering team gravitated toward the flashy stuff — generate an entire course from a prompt, auto-create assessments, AI-powered slide design. Then we talked to actual users. Instructional designers didn’t want AI to replace their expertise. They wanted it to eliminate the tedious parts of their workflow: Reformatting content across different output types (responsive HTML5, PDF, SCORM packages) Generating alt-text for hundreds of images in accessibility-compliant courses Summarizing lengthy SME-provided documents into digestible learning chunks Suggesting quiz questions from existing content (not generating courses from nothing) The takeaway: don’t let engineering excitement drive your AI feature roadmap. Do 10 user interviews before writing a single line of integration code. The highest-impact GenAI features are usually the boring ones. Lesson 2: Prompt Engineering Is a Product Decision, Not an Engineering Task We initially t

Full disclosure: this is directional, not a paper. n=120 tasks, one internal evaluator, not peer reviewed. I work at an LLM infrastructure company. This experiment was done on my own time and is not a company claim. Karpathy's framework classifies tasks by verifiability. Can output be mechanically checked? High verifiability tasks like code compilation and structured JSON extraction are safer because the verifier catches errors. Low verifiability tasks like creative writing are riskier. I wondered if high verifiability tasks are also easier in practice. Can a weaker model do them as well as a frontier model if the verifier catches mistakes? Setup was 120 tasks across four categories. Code unit tests, structured extraction, multi hop reasoning, creative summarization. Three models: Claude Sonnet 4.6, GPT 5.5, local Mistral 3 8B via vLLM 0.6.3. Pass rate for the first two, human rating 1 to 5 for the last two. Results were messy. Code unit tests: Sonnet 4.6 94%, GPT 5.5 91%, Mistral 3 8B 87%. With one retry Mistral 3 hit 95%. That surprised me. I expected the gap to be bigger. Structured extraction: Sonnet 4.6 97%, GPT 5.5 94%, Mistral 3 8B 89%. With retry 96%. Also closer than I expected. But here is where it got weird. Sonnet 4.6 initially scored worse than GPT 5.5 on structured extraction, which made no sense. Turns out our JSON schema had an ambiguous nested array that confused Claude's tool use parser. Fixing the schema brought Sonnet to 98%, but I kept the original numbers in the table because the mistake is part of the story. Your verifier is only as good as your schema. Multi hop reasoning: Sonnet 4.6 78%, GPT 5.5 71%, Mistral 3 8B 51%. Retry didn't help. The model would hallucinate reasoning paths consistently. This is where the capability gap was real. Creative summarization: Sonnet 4.6 4.2 out of 5, GPT 5.5 3.9 out of 5, Mistral 3 8B 3.1 out of 5. Expected. Interpretation: high verifiability tasks seem simpler in the sense that weaker model plus verifier ca

Seems like they have engineered some specific limitations that are widely cited as follows: In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design). Using Claude to develop competing models already violates our Terms of Service, but enforcing this restriction through our safeguards avoids accelerating the actors most willing to violate these terms. Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT). These interventions will not affect the vast majority of coding work. We estimate they will impact ~0.03% of traffic, concentrated in fewer than 0.1% of organizations https://news.ycombinator.com/item?id=48464732 Other comments note how even using the word 'nuclear' in the context of scientific research elicits refusal behavior by the model: https://news.ycombinator.com/item?id=48473302 This makes it seem quite plausible that the model could subtly sabotage any machine learning work (even as false positive). Some suggest this has been happening behind the scenes for a while already, but can anyone confirm that? submitted by /u/AccomplishedCat4770 [link] [留言]

The generation side is shipping fast (TileGym, AutoKernel, KernelEvolve). The validation-side surface for “what the kernel actually did at runtime” has not kept pace. TL;DR In the past nine months, three significant releases have landed for auto-generation of CUDA kernels: NVIDIA TileGym , RightNow AutoKernel, and Meta’s KernelEvolve. Each ships training infrastructure for kernel generation. Validation infrastructure (what the generated kernel actually did at runtime, on a real workload, in a production-shaped environment) has not kept the same pace. eBPF traces are the ground-truth layer that closes the gap. What “validation” means at the kernel level Two distinct validation surfaces: Pre-launch: the generated CUDA C compiles, the PTX assembles, the kernel passes a numerical-equivalence test against a reference. Standard compiler / unit-test territory. Generation frameworks ship this themselves. Post-launch: the kernel ran, returned, took N microseconds, used M registers per thread, hit X cache miss rate, and did or did not serialize the rest of the stream behind it. This is the layer that an eBPF trace plus standard CUDA driver counters can answer for any kernel, generated or hand-written. Auto-generation pipelines do not by default close the post-launch loop. They demonstrate “the kernel works in our test setup”. They do not demonstrate “the kernel does not regress p99 latency on production inference traffic”. What an eBPF trace adds to a generated kernel Once a generated kernel is in a real workload, the same trace surface used for any CUDA kernel applies: launch latency from cudaLaunchKernel , sync stalls from cudaStreamSynchronize , host-side overhead from the dispatcher, host scheduling preemption while the GPU is busy. None of those signals are visible to a generation framework that evaluates kernels in isolation. -- post-launch validation: did the new generated kernel regress p99? SELECT kernel_name , COUNT ( * ) AS launches , AVG ( duration_ns ) / 1 e3 AS

We train a model on handwritten digit classification. 99% accuracy . Then we train the same model on a new task — say, fashion item recognition. We go back and test it on digits. 34% accuracy . It has completely forgotten. Not gradually, not partially — almost entirely. What Just Happened? We trained a CNN on MNIST digits — 99.2% accuracy . After fine‑tuning on Fashion MNIST, it reached 91.1% accuracy . But when re‑evaluated on MNIST, accuracy collapsed to 33.9% . This collapse is catastrophic forgetting : the model’s weights shifted to optimize for the new task, erasing the old solution. Why did training on more data make the model worse at something it already knew? MNIST is handwritten digits (0–9). Fashion MNIST is clothing items like shirts and shoes. Both are 28×28 grayscale images, but the tasks are distinct. Why Does It Happen? The core issue is that the model relies on the same set of weights for both tasks. There is no separation or dedicated memory; every parameter is shared . When training shifts from Task A ( MNIST digits ) to Task B ( Fashion MNIST ), gradient descent simply minimizes the loss on the data it sees at that moment. It has no awareness that Task A ever existed. In the loss landscape, imagine two parabolic bowls: one for Task A and one for Task B. The optimum for Task A lies at θ A ∗ ​ , while Task B's optimum is at θ B ∗ ​ . As training on Task B progresses, the weights θ move towards θ B ∗ ​ . This movement inevitably raises the loss for Task A because its minimum is left behind. The root cause is the shared weight space. Gradient descent is a stateless optimizer; it only follows the current gradient signal. Since the minima for Task A and Task B are far apart, there is no single configuration of θ that satisfies both tasks simultaneously. This is why catastrophic forgetting occurs. Weight space can be visualized as an N-dimensional space, where each axis corresponds to one parameter. Every point in this space represents a full set of wei

I spent a weekend wiring Google's Gemini and Veo APIs into a single app just to feel where the edges of multimodal AI actually are. It turned into a small studio I now use daily, and along the way I learned more about these models from plumbing them than from any paper. Here's the honest technical debrief. Three pipelines, three completely different problems I wanted one prompt box that could do video, image editing, and document Q&A. Naively I assumed they'd share most of the stack. They don't. 1. Image-to-video: the enemy is time, not pixels Generating one good frame is solved. Video is about temporal coherence — frame 13 must agree with frame 12 or you get flicker and identity drift. Modern video models treat the clip as one object in space and time (latent diffusion over a width x height x time volume, with spatiotemporal attention) rather than 120 independent images. Conditioning on a reference image as the first frame is what makes image-to-video feel controlled: you've handed the model a strong anchor and asked it to extrapolate motion, not invent a world. The surprise: native audio sync (Veo 3.1 generating clip + soundtrack jointly) does more for perceived realism than another notch of resolution. A door slam landing on the exact frame the door shuts is uncanny in a good way. 2. Instruction-based image editing: preservation is the hard part Generating is unconstrained; editing must change one thing and preserve everything else. Condition the diffusion model on both the instruction and the source image's latents, cross-attend the instruction to steer only the referenced region, and bias hard toward preserving unedited latents. Push that preservation too soft and the subject's face quietly morphs across edits — the classic 'character consistency' failure that makes or breaks storytelling use-cases. 3. PDF chat: it's retrieval, not a long context The naive 'paste the whole PDF' approach dies on long files (models get lost in the middle ) and costs you the full

Hello Reddit I've been working on QSPR (Quantitative Structure-Property Relationship) analysis for chemical compounds mentioned in the Jean-Claude Bradley Open Melting Point Dataset . Basically the idea is to see how accurate a model can predict melting points of compounds using only topological indices. After some work on the topological indices (feature engineering), each compound was represented by 26 features. I trained a random forest model on the data and got a test r2 score of 0.66 (which is pretty respectable, given the constraints). However, the file size of the model was around 1.23GB. I didn't like it being that big, so I opened up PyTorch to build a custom deep learning architecture that could make predictions as accurately as the random forest but with much smaller file size. After around 2 weeks of research, I build a 270,000 learnable parameter model (1.3-1.4MB according to torchinfo) that got an r2 score 0f 0.6399. Given all this context, I wanted to ask the following question: Should I commit and work on publishing the results, or should I keep working on improving the model? Note: I'm obligated by my university to not give out intricate details of my research before publication, so please forgive me if such details are required for a high quality answer. However, I can give out the metrics achieved by my little deep learning model. Here it is: === Evaluation Metrics (Expected Value) === R² Score : 0.639910 MAE : 41.246754 MSE : 2989.062744 RMSE : 54.672322 NRMSE : 0.083469 MAPE : 11.69% The unit for MAE, MSE, RMSE and NRMSE is Kelvin (K). submitted by /u/AgiGamesYT [link] [留言]

Hi, Niels here from the open-source team at Hugging Face. I've recently relaunched paperswithcode.co as a source for finding the state of the art (SOTA) across various AI domains, from 3D generation to AI agents. This is done by automatically parsing research papers published on arXiv/Hugging Face, enabling leaderboards to be created. See BrowseComp below as an example (a scatter plot and a table are available for each benchmark). - Scatter plot (you can hover over the dots to see the models): https://preview.redd.it/9rz2r3ffcf6h1.png?width=2880&format=png&auto=webp&s=b3f8e7a870802f6ef8227ecc0619e9e1057554b0 - Table: https://preview.redd.it/qoqriddw5f6h1.png?width=2862&format=png&auto=webp&s=a0034574f693847537037013672fb61daf27b16e As you can see, I've added support for viewing evals for closed-source models, too, given that many benchmarks are nowadays dominated by them, like GPT-5.5 and Mythos 5. You can always disable viewing closed-source evals with a toggle or in your PwC settings: https://preview.redd.it/p3k6jt6q6f6h1.png?width=1582&format=png&auto=webp&s=40149e51d6b326a77e53e33baf70d9850b3de365 When you turn them off, here's what the open model leaderboard looks like: https://preview.redd.it/tg42sin36f6h1.png?width=2838&format=png&auto=webp&s=1330a117ae9b4e0ce6d459493ae9e8f64107310a Closed-source papers are treated as regular "papers", although they can be any source, like a blog post (given that PwC supports submitting any source beyond arXiv). See the GPT-5.5 or Mythos 5 papers as examples, with their evals at the bottom. Notice the "closed" tag on their evals. Hence, you could jokingly call these "papers without code". Let me know what you think of this, and whether anything needs to be changed or added! Kind regards, Niels submitted by /u/NielsRogge [link] [留言]

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is...