Website profiles · Technology insights · Alternatives

gsurma.github.io

Greg (Grzegorz) Surma - Portfolio; Machine Learning, Computer Vision, Self-Driving Cars, AI

No paid content found
Visit website
Profile views 2 Outbound visits 10 Normal
Greg Surma - Portfolio Full homepage screenshot

At a glance

Category
Development
Language
English
Paid-content signal
No paid content foundObserved
Registered
Unknown
Hosting
FastlyInferred
HTTPS certificate trusted
PassedObserved
Last updated

Website review

This review may include AI-generated content and is not individually verified. Domain, certificate and other facts come from the observed data on this page.

What is Greg Surma's portfolio?

Greg Surma's portfolio is a personal project showcase for an engineer working across machine learning, computer vision and autonomous driving, presented as a browsable grid of side projects. Each entry pairs a short name and one-line description with tags for platform and technology, so the page reads less like a résumé and more like an annotated workshop shelf: AI cat and dog avatar apps, a CNN explainer, a Jetson-based self-driving toy car, depth estimators for driving, an image tagger, and a handful of Swift/iOS games built around minimax and genetic algorithms.

The mix is the interesting part. Many entries are iOS apps (Swift, CoreML, Metal), while others are Python notebooks or research environments (PyTorch, TensorFlow, OpenCV, Keras, NetLogo). That spread suggests someone who prototypes ideas quickly and then ships them to a consumer platform, rather than specializing in one stack.

Who it suits

  • Recruiters and collaborators scanning for breadth in applied ML and computer vision, especially at the prototype-to-app boundary.
  • Learners looking for concrete project ideas: style transfer, super-resolution, edge detection, lane finding, depth prediction.
  • Developers curious how the same idea looks as a notebook versus an on-device app.

Treat the tags as signposts, not proof of depth; the page gives project names and stacks, not benchmarks or write-ups.

A practical next step

Pick one project matching your own goal and use it as a template. If you want on-device ML, the CoreML-based photo and filter apps show the pattern of training elsewhere and deploying locally; if you want research practice, the depth-estimation and CNN-visualization notebooks show how to frame a narrow, measurable task. For comparison, GitHub hosts many similar project collections, and Kaggle is useful for datasets behind vision tasks.

Which projects in this portfolio are iOS apps built with Swift and CoreML?

The iOS apps that combine Swift with CoreML are mostly visual or playful tools: Kitty Booth (AI cat avatars), Doggo Booth (AI dog avatars), AI Art Filters (style transfer), AI Dreamer (text-to-image), Photo Enhancer (super-resolution), Hashtagram (smart hashtag suggestions), and Edge Detector (holistically-nested edge detection). The excerpt also lists Face Toon (selfies to cartoons) as an iPhone/iPad project, though its tags name PyTorch and computer vision rather than CoreML, so treat it as Swift-based but check its page for the on-device framework.

How to choose among them

  • If you want to see CoreML doing image classification or generation on-device: start with Hashtagram or AI Dreamer. They show how a model's output becomes a usable app feature rather than a demo.
  • If you're interested in camera or photo pipelines: Edge Detector and AI Art Filters are the most direct examples of Swift plus CoreML plus image processing.
  • If you want a simple entry point: Kitty Booth and Doggo Booth are narrow, single-purpose apps, which makes their model-to-UI flow easier to follow than the multi-model projects.

A practical next step is to open the two or three projects closest to your own goal and compare how each one splits work between Swift and the model. For a reader building a photo app, Edge Detector and Photo Enhancer together show the range: one highlights structure, the other reconstructs detail. The portfolio's other iOS entries, such as Doodles, Rocks, Flappy Brain, Islands, Magic Photo Eraser, and Metal Camera, use Swift but list other frameworks (SpriteKit, Metal, TensorFlow, OpenCV) instead of CoreML, so they are useful for Swift patterns but not for CoreML integration.

How does the Jetson self-driving AI toy car project work?

The Jetson self-driving AI toy car is a small robotics project in which a toy car gets a camera and an onboard Jetson computer, and the driving behaviour is learned rather than hand-coded. On the portfolio, Greg Surma lists it as "Jetson Self-driving AI toy car 🧠🚗" with the tags PyTorch, Computer Vision, Machine Learning and Robotics—so the core loop is: camera frames come in, a PyTorch-trained vision model interprets them, and the output drives the car's motors.

What that implies in practice

  • Data collection first. A learned driving model needs examples. Typically you drive the car manually while recording camera frames paired with steering/throttle, then train the network to imitate that behaviour.
  • Vision, not rules. Because the project is tagged Computer Vision rather than lane-detection geometry, the model likely predicts control values directly from images. That contrasts with the lane-finding approach used in his separate "Street Lanes Finder" project, which is a Python notebook on classical CV for self-driving research.
  • Onboard inference. The Jetson is the appeal: it runs the trained model on the car itself, so no laptop tether is needed. The trade-off is that you must keep the model small enough for real-time frame rates, which usually means a compact CNN and quantisation or reduced input resolution.
  • Sim-to-real gap. Indoor lighting, floor texture and battery voltage all shift the images; a model that works on a smooth hallway may drift on carpet. Expect iterative re-recording of data.

Deciding whether to build one

If your goal is to learn the full perception-to-actuation pipeline on real hardware, this is a good shape of project: it forces you to handle latency, power and messy data. If you mainly want to study perception algorithms, a notebook-based lane detector or a depth estimator is faster to iterate on and needs no robot. A middle path is to train in simulation and test a small model on the car.

For related work by the same author, see Greg Surma - Portfolio; for the frameworks involved, the official PyTorch and NVIDIA Jetson documentation are the practical references for training and deployment.

What computer vision techniques are demonstrated in the Face Toon and Photo Enhancer projects?

Face Toon demonstrates a style-transfer / image-to-image pipeline that turns selfies into cartoons. The portfolio lists it as a PyTorch computer-vision and machine-learning project, so the core technique is a learned neural network that maps a photographic face to a stylized cartoon rendering rather than a hand-coded filter. Photo Enhancer demonstrates single-image super-resolution with CoreML on iPhone and iPad, meaning a trained model upscales a low-resolution image and reconstructs detail on-device.

What each project shows

Project Task Technique area Platform signal
Face Toon Selfie → cartoon Learned image-to-image translation / stylization (PyTorch) Computer vision, machine learning
Photo Enhancer Low-res → higher-res image Super-resolution with a trained model (CoreML) iPhone / iPad, Swift

The practical difference is the output goal. Face Toon changes appearance while keeping the subject recognizable; Photo Enhancer preserves content and tries to restore sharpness and missing detail. For a reader building something similar, choose stylization when the aim is a creative avatar, and super-resolution when the aim is cleaning up old or compressed photos.

As a next step, compare these with the portfolio's related entries: Face Generator, AI Art Filters (artistic style transfer, Swift/CoreML) and Image Tagger (a CNN multi-label classifier in Keras). Together they show a consistent pattern: train in Python frameworks, then deploy compact models to Apple devices. For background on the underlying methods, see PyTorch and Apple Developer.

Can I use the ∀llphotos.ai AI stock photo collection instead of a paid stock photo service?

Yes, in the sense that ∀llphotos.ai appears on this portfolio as an AI stock photo collection built with StableDiffusion and Django — but only if AI-generated imagery fits your use case. It is presented as a side project rather than a commercial stock service, so treat it as an alternative for concept visuals, mockups and internal drafts, not as a guaranteed replacement for a licensed library.

Where it fits well

  • Early-stage design and moodboards: placeholder imagery for layouts before you commit to a final asset.
  • Editorial or product pages about AI itself: AI-generated photos read naturally in that context.
  • Low-stakes web content: blog headers, social posts and internal decks where provenance is not critical.

Where a paid service still earns its cost

  • Model and property releases: paid libraries typically document consent for people and places; an AI collection cannot offer that for real individuals or locations.
  • Brand and advertising work: clients often expect clearly licensed, attributable assets.
  • Specific real subjects: named cities, products or events are better served by actual photography.

A practical next step

Before swapping services, list the five images your project most depends on and check each against three questions: does the subject need to be real, does anyone need a release, and would a recognisable AI look undercut the message? If most answers are no, the collection is a reasonable substitute; if any answer is yes, keep a paid subscription for those slots.

For context on the builder's other computer-vision and generative projects, see Greg Surma - Portfolio.

How can I explore the CNN Explainer to understand how convolutional neural networks make predictions?

Open Greg Surma - Portfolio and choose CNN Explainer from the Side Projects grid. It is listed there as a PyTorch computer-vision project about making CNNs explainable, so the portfolio entry is the starting point for exploring how a convolutional network turns an input image into a prediction.

A practical way to explore it

  1. Start with one image and follow the path from input through convolution and pooling layers to the final output.
  2. Watch how feature maps change at each stage: early layers tend to respond to simple edges and colors, while later layers combine them into more recognizable patterns.
  3. Compare several images of the same class and then a different class. The contrast usually teaches more than a single example.
  4. Use the final prediction scores to connect the visual features back to the class the network chose.

The main trade-off is depth versus clarity. A small example network is easier to follow layer by layer, but it will not show the complexity of a production model. If you want a broader set of visual experiments, the portfolio's Features Visualization and GradCam projects address related questions about which image regions influence a prediction.

For a next step, pick one image the network classifies correctly and one it gets wrong, then trace where their feature maps diverge. That comparison is the quickest way to move from "what is a CNN" to "why did this prediction happen."

Related questions

More questions →
What Is Computer Vision and How Can You Use It for 3D Reconstruction?

Computer vision is the field of making software extract meaning from images and video: recognizing objects, tracking motion, and reconstructing 3D structure from 2D pixels. For 3D reconstruction specifically, the practical branch is photogrammetric computer vision — using many overlapping photos of the same subject to recover its shape, camera positions, and surface texture. AliceVision describes itself as exactly this: a "Photogrammetric Computer Vision framework for 3D Reconstruction and Camera Tracking." If you have a camera and a subject you can walk around, you can try this workflow today; if your subject is moving, transparent, or textureless, expect problems.

What computer vision actually does

At a high level, computer vision turns pixel arrays into structured information. The tasks most relevant to 3D work are:

  • Feature detection and matching — finding distinctive points (corners, blobs) in each image and matching them across images.
  • Camera tracking (structure from motion) — estimating where each camera was positioned and oriented, and building a sparse point cloud of the scene.
  • Dense reconstruction — turning the sparse cloud into a detailed surface (depth maps or a mesh).
  • Texture mapping — projecting the original photos back onto the geometry so the model looks real.

Other computer vision tasks — classification, object detection, segmentation — are related but separate. They answer "what is in this image," while photogrammetric computer vision answers "what shape is this, and where was the camera."

Photogrammetric vs. general computer vision

Dimension General computer vision Photogrammetric computer vision
Input Often a single image or video frame Many overlapping photos of one static scene
Output Labels, boxes, masks, tracks 3D points, camera poses, mesh, texture
Key assumption Learned patterns generalize The scene is rigid and visible from multiple angles
Typical tools Neural network frameworks AliceVision, Meshroom, COLMAP, OpenMVG

The distinction matters because photogrammetry fails for reasons general CV does not: not enough overlap, motion between shots, or surfaces with no distinguishable features.

The standard 3D reconstruction workflow

  1. Capture — photograph the subject from many angles with heavy overlap (commonly cited guidance is roughly 60–80% overlap between neighboring shots). Keep lighting consistent and the subject still.
  2. Feature extraction and matching — the software finds keypoints in each image and links them across views.
  3. Camera tracking / structure from motion — it solves for camera positions and a sparse 3D point cloud.
  4. Dense reconstruction — depth maps are computed and fused into a dense surface.
  5. Meshing and texturing — the surface is converted to a mesh and the photos are projected onto it.

Each stage depends on the previous one. If camera tracking fails, everything downstream fails too.

Getting started with AliceVision and Meshroom

AliceVision is the underlying framework; Meshroom is the graphical front end that runs AliceVision's pipeline without requiring you to script each step. The practical entry point for a beginner is Meshroom: you load a folder of photos, start the pipeline, and inspect the resulting 3D model. AliceVision itself is the library you would use if you want to integrate the algorithms into your own software.

A minimal first attempt looks like this:

  • Shoot 30–100 photos of a small object, walking a full circle with overlap.
  • Import the folder into the graph-based interface.
  • Run the default pipeline and watch which nodes complete and which fail.
  • If camera tracking fails, the problem is almost always the photos, not the settings.

Why reconstructions fail

  • Insufficient overlap — gaps between viewpoints leave no matched features.
  • Lighting changes — moving shadows or changing exposure break feature matching.
  • Textureless surfaces — plain walls, glass, and shiny metal give the matcher nothing to lock onto.
  • Motion — anything that moves between shots violates the rigid-scene assumption.
  • Scale ambiguity — a reconstruction from photos alone has no absolute scale unless you add measurements or known reference points.

Where this is actually used

  • Cultural heritage digitization — turning artifacts and sites into archival 3D records.
  • Architecture and surveying — capturing buildings and terrain from drone or ground photos.
  • Game and film asset creation — generating textured models from real objects.
  • Camera tracking for VFX — recovering camera motion to composite CGI into live footage.

If your goal is to learn the pipeline, start with a small, well-lit, textured object and a full circle of overlapping photos. If your goal is a production asset, plan the capture carefully — the reconstruction quality is decided before the software ever runs.

Dell US Official Site: Where to Shop Computers, Monitors, and Get Support

Dell's official US site is dell.com (US storefront under /en-us). Use it when you want to buy a Dell laptop, desktop, monitor, server, or accessory, or when you need order tracking, drivers, warranty help, or technical support. The homepage is organized around a few clear entry points, so you can usually reach the right section in one or two clicks.

Main shopping categories

The top navigation on the Dell US homepage groups products into these sections:

Category What you'll find
Laptops Dell 14S/16S, XPS 13, Dell 14 Plus, Copilot+ PCs
Desktops & All-in-Ones Home and business desktop systems
Monitors Dell Plus ultrawide, Alienware OLED gaming monitors
PC Accessories Keyboards, mice, docks, and desk essentials
Artificial Intelligence AI-ready PCs and related solutions
Servers PowerEdge server lineup
Data Storage Storage products and solutions
Cyber Resilience Security and data protection offerings

If you already know the product type, start from these menus rather than the search bar — the category pages are where current deals and configurations are listed.

Homepage shortcuts for common tasks

The homepage includes a "How can I help you today?" area with direct links for:

  • Show current deals — surfaces today's promotions
  • Find the best laptop for me — guided laptop selection
  • Track my order — order status lookup
  • Get technical support — support entry point
  • Chat with sales — live sales assistance

These shortcuts are the fastest route for order and support questions, which otherwise sit deeper in the site.

Featured products and deals

The homepage currently highlights several promotions and product lines:

  • XPS 13 — described as PCMag's "Best Consumer Laptop," with savings of up to $500 on other select Dell PCs
  • Alienware Aurora gaming laptops — NVIDIA GeForce RTX 50 series graphics
  • Dell 14 Plus — Copilot+ PC with savings up to $370
  • XPS Googlebook — a thin, lightweight XPS model
  • Dell 34 Plus Monitor — 34" curved ultrawide WQHD with single-cable USB-C
  • Alienware 32 4K Monitor — OLED 4K display
  • PowerEdge servers — select models positioned for faster delivery and pricing

Deal amounts and featured models change, so confirm the current price on the product page before deciding.

Support, drivers, and account services

The support area at the bottom of the homepage links to:

  • Support Home — general support hub
  • Drivers and Downloads — driver and software downloads for your model
  • Order Support — help with existing orders
  • Warranty and Contracts — warranty status and coverage
  • Parts and Upgrades — replacement parts and upgrade options
  • My Account — account management

Dell also promotes Dell Care Premium, an add-on for 24/7 expert support that can be added at purchase.

Practical tips before you buy or seek help

  • Have your service tag or order number ready before using order support or drivers — it's the fastest way to get model-specific results.
  • Check the product page for the live price; homepage savings figures are promotional and may have conditions.
  • For business or infrastructure purchases (servers, storage), use the Servers and Data Storage sections rather than consumer categories.
  • Payment options are covered on Dell's separate Payment Solutions page if you need financing details.
What Is CUDA and Why Does It Matter for GPU-Accelerated AI and Machine Learning?

CUDA is NVIDIA's parallel computing platform and programming model that lets software run general-purpose calculations on NVIDIA GPUs, not just graphics. It matters for AI because deep learning and computer vision are dominated by large matrix and tensor operations that a GPU can execute thousands of times in parallel. If you train or run models on an NVIDIA GPU, CUDA is usually the layer doing the heavy lifting underneath frameworks like PyTorch and TensorFlow. If you don't have NVIDIA hardware, CUDA itself won't run — you'd rely on CPU execution or a different accelerator stack.

What CUDA actually is

CUDA (Compute Unified Device Architecture) is two things bundled together:

  • A programming model: you write code that launches many small threads on the GPU, grouped into blocks, and the hardware schedules them across its cores.
  • A platform and toolchain: compilers, libraries, and drivers that connect your program to the GPU.

The core idea is parallelism by the thousands. A CPU has a handful of fast cores optimized for sequential, branchy work. A GPU has many simpler cores optimized for doing the same operation on lots of data at once — exactly the shape of neural network math.

A concrete example

Imagine adding two arrays of one million numbers. A CPU loops through them, a few at a time. A CUDA kernel assigns each element to a thread, so the GPU processes huge batches concurrently. Scale that pattern up to multiplying large matrices, and you have the basic operation behind training a neural network.

Why it matters for AI and machine learning

Modern models spend most of their compute on operations like matrix multiplication, convolution, and normalization. These map naturally onto GPU parallelism, which is why GPU training is dramatically faster than CPU training for anything beyond small models.

Greg Surma's portfolio is a useful illustration of where this shows up in practice. His side projects span PyTorch computer vision work — a CNN explainer, Grad-CAM feature visualization, a Jetson-based self-driving toy car, depth estimation for autonomous vehicles, and a multi-label image classifier — alongside Stable Diffusion and Dreambooth avatar apps. That mix is typical: training and heavier inference on GPU-backed frameworks, with deployment sometimes moving to mobile via CoreML. CUDA is the layer that makes the PyTorch and TensorFlow portions fast on NVIDIA hardware.

Where CUDA sits in a typical workflow

  1. Framework calls the GPU: you write PyTorch or TensorFlow code; the framework dispatches tensor operations to the GPU.
  2. CUDA executes the kernels: the framework's CUDA backend runs the parallel math.
  3. Drivers and libraries bridge the gap: the NVIDIA driver and CUDA libraries (such as cuDNN for deep learning primitives) handle device communication and optimized routines.

You rarely write CUDA C++ yourself unless you're building custom kernels or performance-critical libraries. Most practitioners benefit from CUDA indirectly.

CUDA vs. general GPU computing

CUDA is often used loosely to mean "GPU acceleration," but they aren't the same. The distinction matters when you're choosing hardware.

Dimension CUDA General GPU computing
Vendor NVIDIA only Multiple vendors (e.g., AMD, Intel, Apple)
Ecosystem maturity for AI Very mature; broad framework support Varies by vendor and framework
What you write CUDA C/C++ or framework-level GPU code Vendor-specific APIs or cross-platform layers
Portability Tied to NVIDIA GPUs Depends on the stack chosen

If your goal is running mainstream deep learning frameworks with the fewest surprises, an NVIDIA GPU with a supported CUDA version is the most common path. If you're on other hardware, check that your framework has a working backend for it before committing.

Practical requirements and limits

  • NVIDIA GPU dependency: CUDA code runs on NVIDIA hardware. No NVIDIA GPU means no CUDA execution.
  • Driver and version compatibility: CUDA versions, GPU drivers, and framework builds must line up. A mismatch is one of the most common setup failures — a framework built for one CUDA version may not work with a different installed driver.
  • When CPU or other accelerators suffice: small models, prototyping, and inference on modest workloads can run fine on CPU. Mobile deployment (as in Surma's CoreML-based iOS apps) uses on-device acceleration rather than CUDA.
  • Memory is a real constraint: GPU memory limits batch size and model size; running out is a frequent practical wall.

How to decide if CUDA is relevant to you

  • Training or fine-tuning models of any real size → an NVIDIA GPU with a compatible CUDA setup is the standard choice.
  • Running inference at small scale or learning fundamentals → CPU is often enough to start.
  • Deploying to iPhone/iPad → look at CoreML and on-device acceleration instead of CUDA.
  • Using non-NVIDIA hardware → verify your framework's backend support before assuming GPU acceleration will work.

The short version: CUDA is the bridge that lets NVIDIA GPUs do AI math fast, and it's the default assumption behind most GPU-accelerated PyTorch and TensorFlow workflows — provided your hardware, drivers, and framework versions agree.

What Is AI Charge Capture and How Does It Turn Documentation into Billable Codes?

AI charge capture is software that reads clinical documentation and produces coded, bill-ready charges automatically. Instead of a provider or coder manually translating a visit note into CPT and ICD-10 codes, the system extracts the relevant details from the note and selects codes for review or submission. It fits teams that already document visits in an EHR or EMR and want to reduce missed charges, manual coding searches, and claim delays — MediMobile's Genesis is one example of this category, positioned as an automated medical coding and charge capture solution.

How AI charge capture differs from manual charge entry

Manual charge capture depends on a person remembering to log the encounter, then finding the right codes by hand. That creates three predictable failure points:

  1. Encounters get missed — billable work falls through the cracks when a busy provider moves to the next patient.
  2. Coding takes time — manual searches and reviews slow coders down.
  3. Claims get delayed — late or incorrect charges affect reimbursement.

AI charge capture targets all three by making code selection part of the documentation workflow rather than a separate step after it.

The workflow: from EMR documentation to bill-ready charges

The mechanism MediMobile describes is deliberately narrow: providers document their visits in their EMR, and the system handles the rest. In practice that means:

  • Input: the clinical note the provider already writes during or after the visit.
  • Action: AI coding reads that documentation and generates CPT and ICD-10 code selections.
  • Output: coded charges that are ready for billing, with a charge review step available for coding teams.

The stated result is that documentation turns into CPT and ICD-10 codes "instantly," so encounters are captured before revenue slips away. The provider's job ends at documentation; the coding and charge creation happen downstream.

How the AI selects CPT and ICD-10 codes

The platform describes AI-assisted coding that produces "coded, bill-ready charges from documentation," paired with cleaner charge review and fewer manual searches for coding teams. Two things are worth separating here:

  • Code generation — the system proposes CPT and ICD-10 codes based on what the note contains.
  • Charge review — a human-facing step where coding teams check and clean up those charges before they move toward billing.

That review layer matters because it keeps a person in the loop on code selection rather than treating AI output as final. The source does not specify the model, accuracy rates, or whether any codes bypass review, so treat "autonomous" coding as a spectrum and confirm the review policy with any vendor.

Who each part of the platform serves

MediMobile frames the product around three roles, which is a useful way to check whether a tool fits your team:

Role What the platform provides
Providers Mobile tools to manage patients and capture charges without extra friction
Coding teams AI-assisted coding and charge review with fewer manual searches
RCM leaders Visibility into missed charges, coding progress, and revenue workflows

If your bottleneck is providers forgetting to log encounters, the provider-side capture matters most. If it's coder throughput, the AI coding and review layer is the relevant piece.

Where charge capture connects to the rest of the revenue cycle

Charge capture is one link in a longer chain, and the platform's other features show where it plugs in:

  • MIPS reporting — quality measures are tracked inside the same workflow, so reporting doesn't require a separate data pull.
  • Integrations — connections to EHR, billing, and data workflows, which is what allows charges to move toward billing without re-entry.
  • Reporting and analytics — visibility into missed charges and coding progress for revenue cycle leaders.

The practical takeaway: evaluate AI charge capture by how well it hands off to coding review and billing, not just by whether it generates codes.

Common failure points to check before adopting

The problems MediMobile names — missed encounters, slow manual coding, delayed claims — are the same things to test against in a demo. Ask specifically:

  • Does the system capture encounters from the EMR automatically, or does someone still trigger each one?
  • How are generated CPT and ICD-10 codes reviewed, and who signs off?
  • What happens to a charge the AI can't confidently code?
  • How do charges flow into billing, and what integration work is required?

MediMobile lists "Service Levels & Pricing" and a demo request as the next steps, but the source does not publish prices or plan details, so cost and contract terms have to come from the vendor directly.

What Is Reinforcement Learning and How Does an Agent Learn from Rewards?

Reinforcement learning (RL) is a machine learning approach in which an agent learns by interacting with an environment, taking actions, and adjusting its behavior based on the rewards it receives. Instead of being told the correct answer for every input, the agent tries things, observes outcomes, and gradually favors actions that lead to better cumulative reward. RL is the right choice when you can define a goal as a reward signal but cannot easily label the correct action for every situation—game playing, control tasks, and sequential decision-making are typical fits. It is a poor fit when you already have a large labeled dataset of correct answers, where supervised learning is simpler and more reliable.

The agent–environment loop

Every RL problem reduces to a repeating loop:

  1. The agent observes the current state of the environment.
  2. It selects an action according to its current policy.
  3. The environment transitions to a new state and returns a reward.
  4. The agent uses that reward to update its policy or value estimates.
  5. Repeat until the episode ends or training stops.

The agent's objective is not to maximize the next reward but the cumulative reward over time, which is why RL problems are described as sequential. A move that scores nothing now may set up a much larger payoff later—this delayed-credit structure is what separates RL from ordinary prediction tasks.

How RL differs from supervised and unsupervised learning

Dimension Supervised learning Unsupervised learning Reinforcement learning
Signal Labeled correct outputs No labels; structure in data Scalar reward from interaction
Feedback timing Immediate, per example None Often delayed, cumulative
Data source Fixed dataset Fixed dataset Generated by the agent's own actions
Goal Predict a target Find patterns or groupings Learn a policy that maximizes reward

A useful rule of thumb: if you can write down the right answer for each example, use supervised learning. If you can only say whether an outcome was good or bad after the fact, and the agent's choices affect what happens next, RL is worth considering.

Core concepts you need to recognize

Policy

The policy is the agent's strategy: a mapping from states to actions (or a probability distribution over actions). Training an RL agent essentially means improving this mapping.

Value

A value function estimates how much total reward the agent can expect from a state, or from taking an action in a state. Value estimates let the agent reason about long-term consequences rather than only immediate reward, and many algorithms learn a value function as an intermediate step toward a good policy.

Exploration vs. exploitation

The agent must balance trying new actions (exploration) to discover better strategies against repeating known-good actions (exploitation) to collect reward. Pure exploitation can lock the agent into a mediocre strategy; pure exploration never cashes in what it has learned. Most practical algorithms use a schedule or randomness to manage this trade-off.

Reward design

The reward function is the specification of the problem. A sparse reward (given only on success) makes learning slow because the agent rarely stumbles onto the goal. A misshapen reward can be worse: the agent will optimize exactly what you wrote, including unintended shortcuts. Designing rewards that are dense enough to guide learning but faithful to the actual objective is one of the most consequential and error-prone parts of an RL project.

Common algorithms and tooling

  • Q-learning and Deep Q-Networks (DQN): learn action values and are suited to discrete action spaces, such as choosing among a fixed set of moves.
  • Policy gradient methods: directly adjust the policy's parameters, which extends naturally to continuous actions like steering or joint torques.
  • Actor–critic methods: combine a policy (actor) with a value estimate (critic) to reduce the variance of policy updates.

In practice, these are implemented in Python, often with deep learning frameworks such as TensorFlow or PyTorch, and run against simulated environments so the agent can take many thousands of trial-and-error steps safely and quickly. Greg Surma's portfolio lists reinforcement learning among its topics and includes small game-oriented projects—an α-β Minimax tic-tac-toe variant, a Conway's Game of Life implementation, and a prisoner's-dilemma research environment—which illustrate the same underlying idea: an agent choosing moves in a defined environment and being scored on outcomes. Note that minimax is a classic search technique rather than a learned policy, so it is a useful contrast case: it shows what you can do when the environment's rules are fully known and searchable, versus learning from experience when they are not.

A small concrete example

Consider a grid-world agent that must reach a goal tile. The state is its position; the actions are up, down, left, right; the reward is −1 per step and +10 on reaching the goal. At first the agent moves randomly and accumulates penalties. Over many episodes, value estimates propagate backward from the goal, so tiles near the goal become attractive, and the policy shifts toward the shortest path. If you instead rewarded every step with +1, the agent would learn to wander forever—a direct demonstration of why reward design matters more than algorithm choice in many projects.

Common pitfalls

  • Sparse rewards: the agent never gets a learning signal. Consider shaping rewards or starting with easier versions of the task.
  • Reward hacking: the agent finds a loophole that scores well without solving your real problem. Test the learned behavior, not just the training curve.
  • Unstable training: RL results are sensitive to hyperparameters, random seeds, and environment details. Run multiple seeds and report variance rather than a single lucky run.
  • Too much compute for the question: if a simple rule or supervised model solves the task, RL adds complexity without benefit.
  • Evaluation gaps: performance during training can differ sharply from performance under slightly changed conditions, so evaluate on held-out scenarios.

Deciding whether to use RL

Choose RL when the problem is sequential, feedback is a reward rather than a label, and the agent's actions influence future states. Choose supervised learning when labeled examples exist, and classical search or planning when the environment's rules are fully known and the state space is manageable. If you want to try RL hands-on, start with a small simulated environment and a discrete action space, verify that your reward produces the behavior you intend, and only then scale up the algorithm or the environment.

Website Overview

Identifiable technologies and additional version or configuration signals make the service easier to fingerprint, which may help targeted scanners narrow their checks. Page metadata, canonical configuration and social previews work together to provide more consistent search and sharing presentation.

Domain and Registration

Unknown

DNS and Email

Nameservers are provided by Amazon Route 53, indicating managed DNS hosting. CAA records restrict which certificate authorities are authorized to issue certificates. No CNAME was found; the observed records resolve directly to addresses. No MX record was found. A conventional explicit inbound-mail route is not configured. The lowest observed DNS TTL is 3457 seconds.

TLS and Certificates

The certificate uses an RSA 2048-bit public key, offering broad client compatibility. The server supplied a complete certificate chain. No organization name is present in the certificate; the available fields are consistent with domain validation. The certificate was issued by Let's Encrypt, commonly associated with automated certificate services. The certificate's total validity is about 89 days, consistent with a short renewal cycle.

HTTP and Browser Security

The checked browser-security headers were not detected, leaving fewer explicit browser-side safeguards. CORS permits any origin to read this response. This is common for public resources; sensitive responses need narrower handling. No X-Powered-By header was found, reducing one common source of backend fingerprinting information. The x-cache, x-served-by, via response header indicates a CDN or caching proxy in the delivery path. No obvious internal addresses or debug information were found in the headers.

Technology Stack Analysis

The public page identifies Jekyll 3.10.0, Bootstrap, Google Analytics, Fastly, with exact versions exposed for 1 technologies. These details can narrow vulnerability checks, although exposure alone is not a vulnerability.

Search and Social Sharing

The Generator tag identifies Jekyll v3.10.0, making the publishing system easier to fingerprint. Open Graph is partially configured; og:image is missing. Twitter Card metadata is configured. The title has 22 characters, within a common display range. A meta description is present, with 91 characters.

Hosting and Email

DNSAmazon Route 53
HostingFastly
EmailUnknown
Approximate IP geolocation (inferred; a CDN or proxy may differ from the origin server) United States flagUnited States Inferred 185.199.108.153

Registration details

RDAP / WHOIS

Unknown

DNS records

DNS

TypeNameValueTTLPriority
Agsurma.github.io185.199.108.1533600—
Agsurma.github.io185.199.109.1533600—
Agsurma.github.io185.199.110.1533600—
Agsurma.github.io185.199.111.1533600—
AAAAgsurma.github.io2606:50c0:8000::1533600—
AAAAgsurma.github.io2606:50c0:8001::1533600—
AAAAgsurma.github.io2606:50c0:8002::1533600—
AAAAgsurma.github.io2606:50c0:8003::1533600—
NSgithub.iodns1.p05.nsone.net3457—
NSgithub.iodns2.p05.nsone.net3457—
NSgithub.iodns3.p05.nsone.net3457—
NSgithub.iodns4.p05.nsone.net3457—
NSgithub.ions-1339.awsdns-39.org3457—
NSgithub.ions-1622.awsdns-10.co.uk3457—
NSgithub.ions-393.awsdns-49.com3457—
NSgithub.ions-692.awsdns-22.net3457—
TXTgithub.iov=spf1 a -all3600—
CAAgithub.io0 issue "digicert.com"3600—
CAAgithub.io0 issue "letsencrypt.org"3600—
CAAgithub.io0 issue "sectigo.com"3600—
CAAgithub.io0 issuewild "digicert.com"3600—
CAAgithub.io0 issuewild "letsencrypt.org"3600—
CAAgithub.io0 issuewild "sectigo.com"3600—

TLS and certificates

TLS / SSL

AssessmentNormal configuration
Supported protocolsTLSv1.2、TLSv1.3
Negotiated protocolTLSv1.3
Certificate subject*.github.io
IssuerLet's Encrypt
Valid until2026-12-29T23:44 · Remaining when checked: 67 days
Verification detailsCertificate trust: Passed · Hostname match: Passed

HTTP response headers

HTTP

HeaderValue
content-typetext/html; charset=utf-8
cache-controlmax-age=600
serverGitHub.com
access-control-allow-origin*

Identified technologies

Jekyll 3.10.0BootstrapGoogle AnalyticsFastly

Pages, Search and Sharing

Meta descriptionGreg (Grzegorz) Surma - Portfolio; Machine Learning, Computer Vision, Self-Driving Cars, AI
Canonical URLhttps://gsurma.github.io/
LanguageEnglish (default)
Twitter Cardsummary
googlebot 0 allowed · 0 disallowed
googlebot-image 0 allowed · 0 disallowed
googlebot-mobile 0 allowed · 0 disallowed
msnbot 0 allowed · 0 disallowed
teoma 0 allowed · 0 disallowed
gigabot 0 allowed · 0 disallowed
robozilla 0 allowed · 0 disallowed
nutch 0 allowed · 0 disallowed
ia_archiver 0 allowed · 0 disallowed
baiduspider 0 allowed · 0 disallowed
naverbot 0 allowed · 0 disallowed
yeti 0 allowed · 0 disallowed
yahoo-mmcrawler 0 allowed · 0 disallowed
psbot 0 allowed · 0 disallowed
yahoo-blogs/v3.9 0 allowed · 0 disallowed
All bots 0 allowed · 1 disallowed
  • Disallow/cgi-bin/

User reviews (0)

  • No reviews yet.