How does an OS limit a program capabilities, if it's working directly with the cpu?

Modern CPUs have privilege modes that are used by the operating system lock out certain instructions.  For example in user mode the instructions that modify (raise) the privilege mode or access system resources like the currently configured page tables cause exceptions.  This allows the operating system to decide whether to abort your user process, or to emulate the operation.

Does the OS look into my code before loading it, and sees if I'm using there certain instructions that I'm not allowed to use and if so simply won't execute it?

No, it doesn't have to — the operating system puts the processor into "user" mode whenever it runs user code.  That will activate the hardware's exception mechanism should a privileged instruction be encountered — in user mode, the exception happens instead of executing the privileged instruction (triggering an exception is the execution of that instruction rather than any privileged operation).

Btw, the system call types of instruction used by user mode code to request services from the operating system also activates the hardware exception mechanism.

How is it being managed?

The operating system always runs user code at user mode, aka user level privilege, and generally runs its own code at higher privilege.  These modes inform the processor how to handle privileged instructions.  Most user mode code won't even bother to try to execute privileged instructions as they are useless, but if they do, the hardware exception mechanism kicks in and effectively tells the operating system that this has happened and lets it handle the situation.

In order to run user mode code, the operating system might use a "return from interrupt" instruction, to restart the user code (whether it technically has or hasn't been previously started doesn't matter).  A return from interrupt is a type of instruction that is one way to change the privilege level, while also changing the instruction stream (aka branching); such an instruction itself is privileged, meaning the processor won't allow it in user mode.

When the processor gets an interrupt, it notes some of the critical CPU state — the critical CPU state is that state that it necessarily has to modify to service an interrupt.  Servicing an interrupt transfers control of the instruction stream feed into the CPU, by modifying the program counter aka instruction pointer; on interrupt, the processor effectively makes a sudden branch to the interrupt service routine.  It also makes a sudden change of mode to higher privilege that allows the ISR access to more instructions.  Because these two sudden changes are needed in order to activate the ISR, the hardware will record the prior values for software to use later when resuming the interrupted user mode code.  Thus, the hardware & operating system conspire together to run the OS in at high privilege and the user code a low privilege.

When the user mode program uses a syscall type of instruction (requesting operating system services, like I/O), the same hardware exception mechanism transfers control to the ISR.  When the operating system wants to resume the user mode process, depending on the hardware, it may have to manually advance the program counter of the user mode process across the syscall instruction before resuming — it is as if, to the user mode process, the operating system simulated/emulated the system call.

In addition, all your program's CPU instructions are executed in its own virtual memory space. This basically disables any instructions you code to access or modify system data, other processes data, or kernel data.
Basically, you can mess up your house however you want, but you cannot look into anyone else's houses.

If you want to execute something outside your address space, you have no choice other than calling system routines / services that do it for you - and those check your process' authorization/ privileges.
Staying in the picture above, you need to use a trusted service if you want anything with other people's houses - the OS kernel routines.

This answer assumes you're running a modern multi-tasking operating system (Windows, MacOS, Linux, etc.) on a processor that has specific hardware support for it (ARM, x86, PowerPC, etc.). It draws on the previous answers by Erik Eidt and Aganju.

These systems have developed over the years to become more and more restrictive for ordinary user programs.

On such a machine, your program has no direct access to the physical memory. All memory addresses go through a virtual memory manager. If a particular address has been allocated in your address space, then the address is mapped to a physical memory address. If not, the access fails, and it generates a trap. The operating system will probably terminate your program. The memory manager ensures that no process can ever corrupt another process' memory.

Hardware devices are usually memory mapped. Again, the memory manager blocks any direct access to them. You have to call a driver to do it for you.

There are special machine code instructions used by the operating system to set everything up, and also to implement device drivers. These instructions have different privilege levels required to run them. When your program is launched, it will be marked by the operating system as a user program, which has the lowest privilege. So any attempt to use those instructions to modify or bypass the memory management will be trapped. Again, your program is likely to be terminated.