We investigate the problem of estimating a random variable $Y$ under a privacy constraint dictated by another correlated random variable $X$ . When $X$ and $Y$ are discrete, we express the underlying privacy-utility tradeoff in terms of the privacy-constrained guessing probability ${\mathcal {h}}(P_{XY}, \varepsilon)$ , and the maximum probability $\mathsf {P}_{\mathsf {c}}(Y|Z)$ of correctly guessing $Y$ given an auxiliary random variable $Z$ , where the maximization is taken over all $P_{Z|Y}$ ensuring that $\mathsf {P}_{\mathsf {c}}(X|Z)\leq \varepsilon $ for a given privacy threshold $\varepsilon \geq 0$ . We prove that ${\mathcal {h}}(P_{XY}, \cdot)$ is concave and piecewise linear, which allows us to derive its expression in closed form for any $\varepsilon $ when $X$ and $Y$ are binary. In the non-binary case, we derive ${\mathcal {h}}(P_{XY}, \varepsilon)$ in the high-utility regime (i.e., for sufficiently large, but nontrivial, values of $\varepsilon $ ) under the assumption that $Y$ and $Z$ have the same alphabets. We also analyze the privacy-constrained guessing probability for two scenarios in which $X$ , $Y$ , and $Z$ are binary vectors. When $X$ and $Y$ are continuous random variables, we formulate the corresponding privacy-utility tradeoff in terms of ${\mathsf {sENSR}}(P_{XY}, \varepsilon)$ , the smallest normalized minimum mean squared-error (mmse) incurred in estimating $Y$ from a Gaussian perturbation $Z$ . Here, the minimization is taken over a family of Gaussian perturbations $Z$ for which the mmse of $f(X)$ given $Z$ is within a factor $1- \varepsilon $ from the variance of $f(X)$ for any non-constant real-valued function $f$ . We derive tight upper and lower bounds for ${\mathsf {sENSR}}$ when $Y$ is Gaussian. For general absolutely continuous random variables, we obtain a tight lower bound for ${\mathsf {sENSR}}(P_{XY}, \varepsilon)$ in the high privacy regime, i.e., for small $\varepsilon $ .