you can't take a nonzero value for a.
because if you take a as nonzero then F(infinity) becomes ax+b=infinity which is not possible.
so a should be 0 and b should be 1
No. of points in a circle =n
Since no three points can be collinear so no. of triangle=C(n,3)
No. of line segments =C(n,2)
No. of ways in which 3 line segments can be chosen= C(C(n,2),3)
Probability= C(n,3)/C(C(n,2),3)