Training a ~100M-Parameter Decoder-only Transformer from Scratch
LLM from ScratchImplemented and trained a ~100M-parameter Chinese decoder-only Transformer from scratch, including RMSNorm, rotary positional embeddings (RoPE), and grouped-query attention (GQA). Built both pretraining and multi-turn supervised fine-tuning pipelines, including assistant-only loss masking for conversational SFT.
Pretraining loss decreased from approximately 8.0 to 2.3. Downstream evaluation reached approximately 0.40 accuracy on C3 and 0.55 on XCOPA.
Medical CoT Fine-tuning of DeepSeek-R1-Distill-Qwen-7B
LLM Fine-tuningFine-tuned DeepSeek-R1-Distill-Qwen-7B on approximately 25K medical chain-of-thought samples using QLoRA for parameter-efficient adaptation, exploring low-cost domain adaptation of reasoning-oriented language models.
Validation perplexity decreased from approximately 5.1 to 3.1.
Coding Agent Failure Structure Discovery
Ongoing ResearchInvestigating recurring failure structures in real coding-agent trajectories through failure collection, clustering, and mechanism-level analysis, with a focus on systematic failure modes in long-horizon software engineering tasks.
Counterfactual Calibration of Branch Decisions in LLM Agents
Agent ResearchConducted empirical studies of branch-triggering behavior in SPARK-1.5B agents within the ALFWorld environment. Evaluated 79 states and 237 continuations through counterfactual experiments to investigate relationships among intrinsic trigger signals, branching behavior, and downstream task utility.
Qwen2.5-VL Fine-tuning for Visual Question Answering
Multimodal LLMDeveloped a supervised fine-tuning pipeline for visual question answering, including data preprocessing, data loaders and collators, LoRA / QLoRA configuration, training modules, and experiment setup.
Ship Vibration Monitoring System for Sea-Ice Environments
EngineeringParticipated in the development of a ship vibration measurement system for sea-ice environments, including vibration data acquisition, desktop software, SQL Server-based data storage, system integration, and field testing.