Yes and no! It depends on what kind of object it is that is magnetized and how that object is treated whether or not its magnetic field will diminish or get stronger or change in other ways (like point in a different direction). I can give some practical examples of objects for which the magnetism wearing off is quite common, and others for which it never happens.1) "Permanent" magnets: yes, magnetism does "wear off" spontaneously in these magnets, and the process can be sped up if the magnet is treated in certain ways. Bar, disk, horseshoe, and flat refrigerator magnets are examples of "permanent" magnets you can find in stores. Typical magnetic materials divide themselves naturally into "magnetic domains" -- tiny regions where all the magnetic fields of the atoms are all pointing in the same direction. The magnetic fields are aligned because of an interaction between the electrons of the atoms in the material and their neighbors which favors them to spin in the same direction (see below under electrons). In a fully magnetized magnet, all the domains have their North poles and South poles all pointing in the same direction. This is not the state of the magnet with the least energy, however, as you can demonstrate to yourself with a pair of identical bar magnets. Placing them side by side in contact with both South poles together and both North poles together is quite difficult (but the result is a bigger, stronger magnet). The magnets prefer to be oriented in the opposite direction, with the North pole of one close to the South pole of the other and are happy to stick together like that.
All the little domains in a permanent magnet feel the forces of their neighbors and they would like to alternate. Random thermal fluctuations allow them to do this, but it is unlikely that enough atoms in a single domain will all fluctuate together enough to flip the domain around so its poles point the other way, but it does happen at a small rate. This rate depends on how easy it is to magnetize the material in the first place (how strong those interactions are between the electrons of neighboring atoms). Most permanent magnets will retain their magnetism for many many years, although the decay process can be sped up by heating the magnet or by exposing the magnet to external magnetic fields which oppose its field. Magnetized rock deposits (iron compounds like magnetite) have been found that record the Earth’s magnetic field’s history for at least a hundred million years, so this process is very slow.
2) Magnetic recording media, including cassette tapes, videotapes, and floppy disks. These items’ magnetism most certainly wears off, all by itself, over time, and the process can be accelerated by the handling. Tapes and floppy disks are made of a plastic material that iron or another magnetizable powder is attached to. The grains of powder are magnetized when a magnetic field is applied from the read/write head of the tape player of floppy disk drive. The pattern on these consists of tightly spaced North and South poles. Because of all of the poles so closely packed together, some of them repel others, and gradually over time they will weaken like the permanent magnets mentioned above when placed next to permanent magnets which repel them. Thin tape also has the problem of being wound around spools in the cassette -- the next turn of the tape’s poles affect the tape beneath it. I remember recording a piece of music on a tape -- it was silent at the very beginning and then a very sudden, loud part arrives. After a few years, a faint copy of the loud part could be heard in the silent section one tape-turn earlier, because the poles on the tape in the loud part had magnetized the tape they were next to (sort of backwards of "wearing off", but the net effect is to smear out all the music on my tape over time). Floppy disks also are less reliable years after they are written. Furthermore, the more a tape or a floppy is read, the more it wears out.