You'll need a fishing rod and reel. Look for a rod that's about as tall as you are and has a push button reel. You can also use a cane pole (no reel) and tie your own line to the end.
Gotta have bait too—earthworms rule for most fish. You can buy them, dig for them, or find them lying around after a rainstorm. A net will help you scoop the fish out of the water once you've brought them in. You'll also want a basic tackle box to hold your stuff.