标签：#MachineLearning

Here's a dirty secret of search: "the closest match" and "the most useful result" are not the same thing. Return the mathematically nearest document and you'll often hand someone something technically related and practically useless. Relevance is a harder problem than similarity — and it's where good search is won or lost. Getting that right was the core challenge in the GovernAI Research Atlas , a semantic discovery platform I built to unify research across papers, repositories, and policy. Similarity is not relevance Semantic search gives you a superpower: embed everything into vectors and find items close in meaning , not just wording. But raw nearest-neighbor retrieval has a blind spot. The vector-closest result might be a tangential paper that happens to share vocabulary, while the genuinely useful one sits slightly further out. Distance in embedding space is a proxy for relevance — a good one, but not the whole story. If you stop at "closest vector," your search is clever and still frustrating. Ranking on top of retrieval The Atlas runs ChromaDB vector search with Sentence-Transformer embeddings across sources like OpenAlex and GitHub — that's the retrieval layer, the "what's semantically near this query." On top of it sits a custom relevance score that decides what actually surfaces first. That two-stage shape is the pattern behind every search system worth using: Retrieve broadly by meaning. Vectors pull in the semantically-relevant candidate set, fast, across a large and varied corpus. Rank deliberately. A custom scoring layer reorders those candidates by what's actually useful — because the job isn't to return related results, it's to return the right one first. Unify the sources. Papers, code, and governance material ranked into a single relevance-ordered experience, so discovery crosses formats instead of siloing them. Why this is the interesting part Retrieval gets the attention; ranking gets the results. Anyone can wire up a vector database and get "se

Nothing happened. That is the strange part. No deploy. No pull request. Nobody touched the prompt. Your agent ran the way it always ran on Friday, and it runs on Monday, and every dashboard is green. Then a ticket comes in about an answer nobody on your team would have written, and you go looking for the change that caused it, and there is no change on your side. There was a model upgrade. It is the only change to your system that you did not make, cannot find in your own git history, and usually cannot roll back on your own schedule. It is also the one most likely to be announced to you as good news. Why a model upgrade does not look like a bug Because it is not one. The new model really is better. Better on reasoning, better on code, better on the evaluations the lab published beside it, and probably better on yours too, if what you measured was the average. Better and same are different words. Your product was not built on the average. It was built on a specific set of behaviours you watched, liked, and then quietly encoded into everything downstream: how long the answers run, how much the thing hedges, which tool it reaches for first, what it does when a request is vague. None of that appears in release notes. All of it can move. And when it moves, nothing throws. There is no stack trace for "this answer is now worse in a way a customer will notice." Your tests keep passing, because your tests check that the JSON parses and the fields are there, and the JSON still parses and the fields are still there. The three things that actually move Shape. Answers get longer, or shorter, or start opening with a summary they never used to open with. Harmless, right up until something downstream was written against the old shape. Tool choice. The agent develops a new favourite first move. It takes six calls to do what used to take three, or it stops calling the tool you built for it because it has decided it can answer from memory. This one usually reaches the bill before it

I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed. How the long context works: the most recent 2048 tokens stay in fp16 like a normal KV cache. Everything older gets compressed to 1 bit and written to disk, about 320 bytes per token, so 1 million tokens of history is roughly 320 MB on disk. From the start the model was trained to retrieve from that disk cache, up to 100M tokens. Due to a limited budget it wasn’t trained to reason over those tokens, only retrieve and answer from them. Language modeling quality of the base model, measured on held-out English web text that was never seen in training (educational web pages, 2,048 token windows): cross entropy 3.15 nats per token, perplexity 23.3, 0.99 bits per byte. The vocabulary is also not a normal embedding table. Every token is a fixed 512-bit code, 8.4 MB for all 131k tokens, zero trained parameters. I tested it on WordSim-353 (human word similarity ratings): my table scores 0.619 Spearman correlation vs 0.029 for random codes. Test script is in the repo. Some outputs (settings included so you know I’m not cherry picking, all reproducible from the repo): “Explain photosynthesis in two sentences.” (greedy) Photosynthesis is a process in which plants convert sunlight into chemical energy, which is then used to produce oxygen and other chemicals. This process is called photosynthesis. “Write a short poem about the sea.” (temp 0.25, top-k 30, rep 1.15, seed 2) The waves had swept over, and they were crashing against each other like rocks on top of one another. The wind was strong but not very gentle as it tried to push them towards their destination without lifting any weight or pulling away from those who stood in front of her with a stiff smile that made the sea look even more majestic than before. “What is the serial number of devic

Taking the right medication at the right time is more than just a routine—it's a critical part of healthcare. However, for the elderly or those with complex prescriptions, "pill fatigue" is real. Mistakes happen. In this tutorial, we are diving deep into Computer Vision , Edge AI , and IoT to build a real-time pill identification and reminder system. We will leverage YOLOv8 for multi-pill detection and semantic segmentation, deploy it on a Raspberry Pi , and use MQTT to trigger physical alarms or notifications. Whether you are looking to master real-time object detection , explore embedded AI implementation , or build a life-saving IoT device , this guide has you covered! The Architecture: From Vision to Action 🏗️ The system follows a classic Edge-to-Cloud (or Edge-to-Local) pattern. The Raspberry Pi acts as the brain, processing image frames locally to ensure privacy and low latency. graph TD A[Raspberry Pi Camera] -->|Video Stream| B[OpenCV Preprocessing] B --> C{YOLOv8 Engine} C -->|Detection/Segmentation| D[Logic Layer: Check Schedule] D -->|Match/Mismatch| E[MQTT Broker] E -->|Publish Topic| F[Physical Alarm / Buzzer] E -->|Status Update| G[Mobile App/Dashboard] D -->|Log Data| H[Local Database] Prerequisites 🛠️ To follow along, you'll need: Hardware : Raspberry Pi 4B/5 (8GB recommended), Camera Module (V2 or HQ). Tech Stack : YOLOv8 : For state-of-the-art segmentation and detection. OpenCV : For image manipulation. Paho-MQTT : For the messaging protocol. Ultralytics : The framework powering our model. Step 1: Training the YOLOv8 Segmentation Model While YOLOv8 is famous for object detection, we use Semantic Segmentation here to precisely calculate the area and shape of pills, which helps distinguish between very similar-looking tablets. from ultralytics import YOLO # Load a pretrained model model = YOLO ( ' yolov8n-seg.pt ' ) # Train the model on our custom pill dataset # Assume we have a 'pills.yaml' defining classes: 'aspirin', 'vitamin_c', etc. results = mo

Gen AI Academy APAC — Track 1 (AI Agents with Gemini, ADK, and Cloud Run) Why I built this I lead a customer service team of 25 agents at Amazon, handling both buyer-side and marketplace seller support. A big part of my job is reviewing escalated cases — calls or chats where a customer asked for a supervisor — and figuring out why they escalated in the first place. Was it a policy gap? A training issue? A system limitation nobody flagged? Right now, that review is manual. Every escalation gets read, tagged, and turned into a coaching note by a human — usually me, or one of my leads. It works, but it doesn't scale well, and patterns across dozens of cases are easy to miss when you're reviewing them one at a time between everything else on your plate. So for Track 1 of the Gen AI Academy APAC program, I built an agent that does the first pass of this analysis automatically: read an escalation summary, classify the root cause against a standard taxonomy, flag whether it looks like a repeat pattern, and draft a coaching note — the same way I would, just faster and more consistently. What it does The agent takes a case summary like this: Customer requested a refund for a damaged item outside the standard return window. Agent denied it citing policy; customer says a rep last month approved a similar exception for someone else. And returns a structured analysis: { "root_cause_category" : "policy_misapplication" , "severity" : "medium" , "is_likely_repeat_pattern" : true , "pattern_reasoning" : "Inconsistent policy application across agents suggests a training or documentation gap rather than an isolated error." , "coaching_note" : "..." } It's built on Google's Agent Development Kit (ADK) with Gemini as the underlying model, and deployed as a live service on Cloud Run . The agent has one tool — a lookup function for the standard root-cause taxonomy — which keeps the categories consistent and easy to update without touching the core prompt. For batch review, I also built a

So got selected for a research internship at MSR, how good is the quality of work and how useful is it to move to Applied sciences or research sciences position at other FAANG companies after the internship. And any perks and other benefits that interns get during microsoft internship? Any tips will be appreciated. Specifically to get into AS at amazon , does this boost my chances? I'll be joining as an SDE-1 at amazon after 6 months so planning to apply internally once I join. So what else should I do to improve my chances to go to AS. submitted by /u/Fuzzy-Pool2415 [link] [留言]

LLMs are too verbose and with a black box model the only things you control are what goes in and how you tell it to write back. Yesterday Claude Code shipped a "concise output style" where Claude keeps things short. We already have a paper out about this! We tested both channels, shortening the input prompt versus telling the model to output answer shorter, on the same questions across five reduction levels, and scored cost, accuracy, and whether the shortened text still matched what the model would have said unconstrained. We also evaluated GPT-4o, GPT-5.4, Claude Haiku 4.5, Claude Sonnet 4.6, Qwen2.5-VL-7B, Qwen3.5-9B, DeepSeek-R1-Distill, Gemma-4-E4B, and Kimi-K2.6 + benchmarked on five short answer datasets + a eleven-language output run (English, German, Spanish, French, Swahili, Chinese, Japanese, Russian, Bengali, Thai, Telugu) + a longer-form summarization test. (1) Shortening the output saved money while keeping accuracy about the same, about 1.5x cheaper on average and up to 3x in the best case across the API models. It worked across languages too! (2) Shortening the input prompt did the opposite. It cost up to 96% more on the worst benchmark, because the model just answers longer to fill in for what you cut and accuracy drops. You pay more and get worse answers :( (3)Output tokens cost more than input tokens, so prompting for fewer output tokens would save costs with short single turn tasks (4) When the shortened output is correct, about half the time the text no longer matches how the model would have reasoned without the constraint. Which is probably fine if you only care about the final answer With providers now offering concise options, we can't see how they're charging for it, so we don't know if it actually saves you cost. But if you control the prompting yourself via the API, you actually do save!! Paper https://www.alphaxiv.org/pdf/2606.24083v1 Code + data https://github.com/danielle34/cavewoman submitted by /u/ibubbles34 [link] [留言]

I have built an on-prem GPU cluster, 8 nvidia 16GB GPU's and 256GB CPU RAM, 50TB HDD and several TBs of SSDs. I have used it, and currently use it, for ML/AI research. But that research is not constantly running jobs, sometimes I use it heavily and other times it's idle. I was considering just letting people with qualified use cases run jobs on it SLURM style. I don't know if its enough compute to be useful really. Let me know if it's something you'd be interested in using for your research? what would you actually run in ~200 GPU-hours on 8x16GB cards? I've found it can handle RLVF pretty well, and I have pretrained models up to 500M parameters on it (research size). But obviously it's no stargate cluster submitted by /u/redwat3r [link] [留言]

What is up with the EMNLP prices? What is the actual price for attending as a student with one accepted paper? If I register now in August, is it $350 or $550? Congratulations to everyone accepted! https://preview.redd.it/to16g93h7rkh1.png?width=667&format=png&auto=webp&s=566162320e8adc161ab3a3772988c6ea64d8be6d submitted by /u/No_Sky9786 [link] [留言]

Today I'm releasing Pythonaibrain-NLP 0.2.0 , the latest public release of my Python NLP framework. The package is now available on PyPI, and the complete source code, documentation, architecture notes, examples, and tests are available on GitHub. PyPI: https://pypi.org/project/Pythonaibrain-NLP/ GitHub: https://github.com/DivyanshuSinha136/Pythonaibrain-NLP Install it with: pip install pythonaibrain-nlp Why another NLP framework? Pythonaibrain-NLP was built around a different idea. Instead of making a transformer the center of everything, I wanted to build a more structured NLP system where understanding, dialogue state, retrieval, and generation are explicit components of the architecture . The current system combines: Neural intent classification Slot filling Dialogue context Retrieval-augmented responses Neural language generation A controllable NLG architecture Standalone NLU and NLG APIs The goal isn't to replace every modern NLP architecture. The goal is to provide a structured, understandable, trainable NLP pipeline that can be integrated into Python applications. The architecture The core pipeline is: User Input │ ▼ ┌─────────────┐ │ NLU │ │ │ │ Intent │ │ + Slots │ └──────┬──────┘ │ ▼ ┌─────────────────┐ │ Dialogue State │ │ + Context │ └────────┬────────┘ │ ┌───────┴────────┐ ▼ ▼ Function/API RAG Dispatch Retrieval │ │ └───────┬────────┘ ▼ ┌─────────────┐ │ NLG │ │ SC-LSTM │ └──────┬──────┘ │ ▼ Response This separation makes each stage independently accessible and easier to experiment with. NLU The NLU subsystem uses a joint neural architecture for: Intent classification + slot tagging The model is designed to understand both what the user wants and which pieces of information are present in the input . For example, a request such as: "Book a flight to Delhi tomorrow" can be represented through an intent together with structured slot information rather than treating the entire sentence as an opaque classification problem. This structured representation ca

Classic Machine Learning Through the Eyes of an SRE — Part 7 Picture a client health metric that has been flat at 2 out of 10 for six months. Ask PCA to compress your client-health data and that metric will contribute almost nothing to the directions PCA decides to keep. Not because PCA is broken. Because PCA treats variance as importance, and a signal that barely moves contributes almost no variance. Reduce the data far enough and the independent information it carried is simply not there anymore. But a CSAT frozen at 2/10 is not noise. It is a crisis nobody is escalating. And after compression, it may no longer be available to anything downstream. That is the bet, and in ops data it is frequently wrong. The critical signals are often the quiet ones. There is a cheaper version of the same failure that catches most people first. PCA measures variance in whatever units your features happen to be in, so a metric ranging from 0 to 10,000 can dominate one ranging from 1 to 5 purely because it is bigger. Standardize before you compress, or your first principal component may just be an elaborate way of saying "ticket count." Same class of bug as unscaled features in K-Means and SVM, and it fails just as quietly. What PCA actually is Third answer-finding strategy in the unsupervised set, using the same shorthand as the last two articles. K-Means SEARCHES: iterate and hope. DBSCAN DEFINES: declare a rule and traverse. PCA SOLVES: an eigendecomposition or SVD gives a direct solution rather than an iterative local search. No convergence to babysit, no restarts, no local optima to escape. Two caveats on the word "direct," both worth knowing. Many libraries will use randomized SVD on large matrices, which is approximate and stochastic. And even with an exact solver, eigenvectors are only defined up to sign, so a component can come back inverted between runs or across implementations. The variance explained is identical either way, which is precisely why nobody notices. Hold ont

How many times have you stared at a plate of Gong Bao Chicken or a complex Mediterranean salad and wondered, "How many calories are actually in here?" Traditional calorie tracking apps are tedious, requiring you to manually weigh ingredients and search through messy databases. But with the rise of multimodal AI , specifically the GPT-4o Vision API , we can now transform a simple photo into a detailed nutritional breakdown in seconds. In this tutorial, we are building a Computer Vision Nutrition Engine that leverages GPT-4o to identify ingredients, estimate portions, and calculate macronutrients with surprising accuracy. By using Few-shot Prompting and structured data validation with Pydantic , we’ll solve the age-old problem of identifying "hidden" ingredients in complex cuisines. Whether you're interested in AI for health or mastering multimodal LLM pipelines , this guide is for you! The Architecture 🏗️ The system logic is straightforward but powerful. We take an image input, process it through the GPT-4o vision model using a specialized system prompt, and enforce a strict JSON schema output for our frontend to consume. graph TD A[User Uploads Food Image] --> B[Streamlit Frontend] B --> C{FastAPI/Python Logic} C --> D[GPT-4o Vision API] D --> E[Few-Shot Prompting Strategy] E --> F[Pydantic Structured Output] F --> G[Calorie & Nutrient Dashboard] G --> H[User Review & Log] Prerequisites 🛠️ To follow along, you'll need: Python 3.9+ OpenAI API Key (with GPT-4o access) Libraries : openai , streamlit , pydantic , pillow Step 1: Defining the Data Schema with Pydantic To make our engine reliable, we can't just accept raw text from the AI. We need structured data. We’ll use Pydantic to define exactly what a "Nutrition Report" looks like. from pydantic import BaseModel , Field from typing import List class Ingredient ( BaseModel ): name : str = Field ( description = " Name of the ingredient identified " ) estimated_weight_g : float = Field ( description = " Estimated weight

In our inaugural issue , we established that the future of enterprise AI lies not merely in raw model parameters, but in the architectural paradigms—specifically Graph Neural Networks (GNNs)—that capture relational intelligence. However, the most sophisticated architectural decision is rendered obsolete if the deployment infrastructure introduces prohibitive latency. At Informatiqs, we emphasize that model deployment is fundamentally an operations research problem. As we transition from batch-processed predictions to real-time Generative AI and dynamic Machine Learning on Google Cloud Platform (GCP), we confront the inherent friction between compute elasticity and system responsiveness: the notorious "Cold Start" problem. In this issue, we dissect the mathematics of serverless inference, the orchestration of Cloud Run and Eventarc, and how minimizing initialization latency is the ultimate enabler for high-frequency, event-driven enterprise intelligence. 1. The Mathematical Anatomy of the Cold Start To engineer a solution, we must first formalize the problem. In a serverless architecture (scale-to-zero), infrastructure scales dynamically with demand. The total response time for an inference request can be understood as a composite of three phases. First, the baseline network latency. Second, the actual inference time—the computational effort of the model itself. The critical variable, however, is the conditional penalty phase. If a serverless container has scaled to zero, the system must endure the time required to provision new compute resources and the heavily taxing process of loading massive neural network weights into memory. If the container is already 'warm', this penalty is completely bypassed. We can model the probability of encountering this cold start using queueing theory. Assuming incoming inference requests arrive as a stochastic process, the likelihood of a cold start is determined by the mathematical relationship between the frequency of incoming requ

In the current epoch of Artificial Intelligence, the industry remains singularly preoccupied with the "Model" — obsessing over the raw parameter scales of the latest LLMs or the specific benchmark performance of a new transformer variant. However, at Informatiqs, we shift the lens. We recognize that sustainable enterprise value is rarely derived from the model in isolation; instead, it emerges from the high-stakes architectural decisions and systemic orchestration that define its environment. As we launch our inaugural edition, we dissect a critical technological nexus: the convergence of Graph Neural Networks (GNNs), Generative AI, and the industrial-grade infrastructure of Google Cloud Platform (GCP). We argue that for complex enterprise datasets, the transition from flat vector embeddings in latent space toward non-Euclidean, graph-based relational intelligence is the primary differentiator for the next generation of resilient AI applications. 1. The Scientific Foundation: Exploiting Relational Inductive Bias Traditional Deep Learning architectures, such as Convolutional Neural Networks (CNNs) for images or Transformers for text, primarily operate on data structured as sequences (Euclidean space). While exceptionally powerful, these structures often fail to capture the topological nuances of real-world systems like supply chains, molecular structures, or fraudulent transaction webs where data is inherently non-Euclidean. Graph Neural Networks (GNNs) provide a framework for learning from data represented as nodes and edges. Unlike standard neural networks that process inputs in isolation, GNNs utilize a Message Passing paradigm. In this process, a node's internal representation is iteratively updated by aggregating information from its immediate neighbors. Instead of looking at a data point as a single row in a database, the GNN looks at who that data point "talks to" and how those connections define its identity. By utilizing Graph Attention mechanisms, we can fu

gex.live has two halves. The terminal measures where SPX options dealers are positioned, every second, from the tape. The Lab is the half that asks the uncomfortable question: does any of that predict anything? What it is A browser-side conveyor with three stages and a credit meter. Compile. You describe a rule in plain text — "short the first touch of the put wall when net gamma is below the 20th percentile" — and the compiler turns it into a deterministic rule over the archive's fields: flip, walls, hold band, gamma percentile, DEX/VEX/vanna/charm per strike, time of day. Compiling is free. If the text is ambiguous the compiler says which part, instead of guessing. Backtest. The rule runs against the full session archive — 1,000+ finished SPX days, every one of them public at gex.live/sessions — with a fixed out-of-sample split. One credit per job; a job that fails refunds itself. Quant optimize. Optional. A LightGBM pass over the same feature store to see whether there is structure the hand-written rule missed, reported as out-of-sample AUC plus feature importance, not as a new "signal". The heavy part (DuckDB + LightGBM) runs in a scale-to-zero container that reads snapshots over HTTPS from the public archive. It depends on no machine and on no private data, which is the point: you are testing against the same files anyone can download. The honest-stats rule Every verdict comes with its baseline. "Your rule made 3% in-sample" means nothing next to "the unconditional drift over the same days was 2.8%". The report shows both, shows the out-of-sample half separately, and refuses to produce a headline number from the in-sample half. Most rules do not survive this. That includes our own: the site's own directional levels were tested three separate ways across the whole archive and none held out of sample — which is why the terminal sells measurement and not signals, and why the Lab exists at all. The free Idea Feed Next to the conveyor sits a rail of rule-shaped idea

Most speech-to-text is benchmarked on audio that looks nothing like a WhatsApp voice note. The standard evaluation sets are read speech, broadcast news, or recorded interviews: single speaker, decent microphone, one language, quiet room, speaker aware they are being recorded. A WhatsApp voice note is close to the opposite on every axis. I have spent a while building around this, and the gap turned out to be wider than I expected. Acoustics Phone held at arm's length while walking, in a car, in a kitchen, on a street. Distance-to-mic varies wildly within a single recording , which breaks a lot of assumptions about consistent gain. Then there is the codec. Voice notes are Opus at low bitrate — efficient, but it discards exactly the high-frequency detail that helps disambiguate fricatives. /s/ versus /f/ versus /th/ get genuinely harder, and those distinctions carry real meaning. Register Conversational, not read. False starts, self-corrections, filler, trailing off mid-sentence, and long pauses that are not sentence boundaries — someone thinking, or getting distracted. Punctuation inference is much harder here than on read speech. And punctuation is most of what makes a transcript skimmable rather than a wall of text. A perfectly accurate word sequence with no paragraph breaks is close to useless if the point was to let someone read it faster than listening. Language This is the one that surprised me most. Voice notes are heavily code-switched. People drop English technical terms into Urdu, Hindi, Arabic, Spanish sentences constantly — not as an edge case, as the default register for a huge number of speakers. If you force a single language selection up front, you mangle every mixed utterance. Auto-detection is not a convenience feature in this domain. It is a correctness requirement. Length distribution Most notes are 5–45 seconds. Very little context to work with, and per-request overhead dominates if you architected for long files. Batching strategies that make sen

Why plain k-fold silently overfits your trading model — and the 4-line fix that stops it. The Problem With k-Fold in Time Series Financial data is sequential. k-fold shuffles rows, so a training row from 2 PM Tuesday sits next to a test row from 10 AM Monday. Worse: triple-barrier labels overlap . A label at bar t looks 6 bars into the future; a training row at t+2 "knows" part of that future. The model leaks. V1's history is full of "HIGH overfit" verdicts — train AUC high, test AUC flat. Plain TimeSeriesSplit is only marginally better; it still lets adjacent windows bleed into each other. Purged + Embargoed CV For each test window [t0, t1] : Purge any train row whose label window overlaps the test window. Embargo max_training_horizon bars after the test window — drop those too. Overlapping labels are not i.i.d. Purging + embargoing makes the split honest. def purged_embargo_split ( n , n_splits = 5 , embargo_frac = 0.02 ): idx = np . arange ( n ) fold = np . array_split ( idx , n_splits ) splits = [] for i in range ( n_splits ): test = fold [ i ] emb = int ( len ( test ) * embargo_frac ) lo , hi = max ( 0 , test [ 0 ] - emb ), min ( n , test [ - 1 ] + emb + 1 ) train_mask = np . ones ( n , bool ); train_mask [ lo : hi ] = False splits . append (( idx [ train_mask ], test )) return splits Tune Only When You Have Enough Optuna once "won" a validation set with only 4 decisive rows — statistically meaningless. Rule: never tune when the decisive (non-abstained) validation rows are below ~30–50. Widen the date range or symbol basket first; don't trust the trial. Three-Way Split, Always train (fit) → validation (early stop + HP select) → disjoint calibration set (sigmoid/ isotonic) → test (untouched, final score only). V1 sometimes conflated validation and calibration. Keep them separate. The Promotion Gate Log every trial's train/val/test gap, not just the winner's test score. Promote only if replay AND shadow (≥1 live session) both beat baseline on buyer metrics : 1.5x

Real integration notes for wiring NIFTY ML models to live broker data via Dhan. Research/ paper-trading context — not a live-trading recommendation. Why Dhan Dhan's API exposes direct option-chain access — exactly what an options-ML system needs: POST /optionchain — full chain for an underlying POST /optionchain/expirylist — available expiries Fields: security_id , last_price , volume , oi , previous_oi , implied_volatility , top_bid_price , top_ask_price , and greeks (delta/theta/gamma/vega) Security IDs are stable: NIFTY = 13 (IDX_I) , BANKNIFTY = 10001 (IDX_I) . The Pipeline Shape A research dashboard pulls live chain + underlying, runs the trained XGBoost model on each new 15-minute bar, and displays: side score (CE/PE alignment) gate state (entry ready / blocked) contract quality scores a doctrine/backtest report Keep the inference path separate from the execution path . The dashboard shows; a permissioned, human-approved module places orders. Paper Trade First The DhanLiveTrader pattern: load the model, predict on each new bar, place long orders with configurable SL/TP (default 1.0 ATR SL, 2.0 ATR TP), and run in paper mode first . Only after stable out-of-sample + paper evidence should any execution module even be considered. { "client_id" : "YOUR_DHAN_CLIENT_ID" , "access_token" : "YOUR_DHAN_ACCESS_TOKEN" , "is_paper_trade" : true , "nifty_symbol" : "NIFTY" , "quantity" : 50 , "max_trades_per_day" : 3 , "sl_atr_mult" : 1.0 , "tp_atr_mult" : 2.0 } The Hard Part: Stops A known footgun: using a Stop-Loss Limit (SL-L) order with price = sl − 0.05 means it won't fill if price crashes through the stop. Prefer SL-Market for the protective stop. Execution quality is its own research topic — don't bolt it on at the end. Honest Status The ML side of this stack showed real directional skill (60.5% top-decile accuracy) but the fixed-SL backtest was still unprofitable (PF 0.53). A dashboard that displays an honest "RESEARCH / PAPER" status is worth more than one that hid

Lessons from a real rebuild of an options-buyer prediction system. No profit claims — just the architecture that fixes the chronic bugs of V1. The Core Mistake in V1 V1 asked one XGBoost model one big fuzzy question: "CE ya PE?" — directly from raw CE/PE premium data. Premium is a transformed signal (underlying move × delta × gamma × IV × theta × spread × strike distance × liquidity). The model learned noise as much as signal. Concrete evidence from the research logs: Balanced accuracy stuck at 51–61% for months — hyperparameters were never tuned ( lr=0.02, depth=3 defaults used throughout; Optuna existed but was never run). A partition bug ( iv_change_1d shift inside single-row groups) silently zeroed a whole feature for the entire history. A rollup config flag compressed 15-minute bars into 1 row/day, destroying 760× of training volume (387 sequences instead of 295K+). Live paper trading: 31.6% win rate, −₹90.3k PnL , entry confidences only 55–64%. V2 Principle: Split the Question underlying mechanics --> side, range, ETA, invalidation option chain scanner --> is the buyer contract worth paying for? XGBoost (many heads) --> thin calibrated learner on clean mechanics Rule: underlying decides side; option contract decides execution eligibility. CE/PE premium is validated against, never learned as, direction. Many Shallow Heads, Not One Deep Model Instead of one CE/PE answer, V2 trains separate narrow heads: underlying_up/down_touch_{15,30,60}m ce_1p3x / ce_1p5x / ce_2p0x and pe_1p3x / pe_1p5x / pe_2p0x (SEPARATE CE and PE) no_trade_quality This single change removes most of the CE/PE confusion V1 fought for months. The Shallow Regularized Grid (the actual fix for overfit) learning_rate = 0.015 – 0.035 n_estimators = 800 – 2000 ( early stop ) max_depth = 2 – 3 min_child_weight = 12 – 40 gamma = 0.1 – 2.0 subsample = 0.65 – 0.90 colsample_bytree = 0.55 – 0.85 reg_alpha = 0.5 – 3.0 reg_lambda = 6.0 – 20.0 scale_pos_weight = min ( neg / pos , 8.0 ) V1's intraday head ha

How a theory of leftover surprise changed a memory layer Richard Emate Richard Emate Richard Emate Follow Aug 18 How a theory of leftover surprise changed a memory layer # python # ai # llm # opensource Add Comment 9 min read