It can be really easy - if you're in an appropriate boat. I would recommend a recreational kayak or sit on top (SOT) kayak for your first few attempts. These are wide, stable boats - that are almost difficult to flip.
You've got the right idea with the life vest - but you don't just want to "have" it, you need to wear it. And if you're in an area where the water and air are cooling off already, you need to be dressed appropriately. Read the article I linked below to get a full understanding of how you can prepare yourself.
The biggest mistake first time paddlers make, and this goes especially for the guys, is they try to muscle the paddle through the water. This is completely the wrong way to do it, and for good reasons: although you might have big biceps, you're going get tired and sore in a hurry. Kayaking technique requires using large muscle groups - and in this case, your core muscles are what are needed. Rotating from the hips, you'll have better endurance and a more efficient forward motion if you rely on your abs to do the work. Women naturally have less upper body strength, and seem to find this rotation idea a little more quickly.
What can go wrong? If you're in the wrong boat, in bad conditions, not wearing the life vest, not dressed for immersion: death can happen. It's possible. But, since you're here asking for advice, you obviously are smart and realize that there are risks and you are willing to do something to minimize them.
Dress right, don't go alone, best to rent boats or take a class from a paddling shop to get you started properly. Paddling is a blast - and people of all ages can do it in some form or another. Be safe, and have fun!