Conceptualisation

The process of conceptualisation can be represented as the following ordered triple:

where:

• is the domain space, which is the set of all the elements.
• is a maximum set of states possible for all n-ary relations in .
• conceptual relations on the domain space .

Ontological Commitment

In a logical language , with a vocabulary , we fix an intensional interpretation by means of a structure , where is the conceptualisation and is a function assigning the elements of to constant symbols of , and elements of to predicate symbols of . This is the intensional interpretation of ontological commitment for . We use here, because is assigned to either or with function . The function assigns meanings to vocabulary symbols by mapping constants to domain entities and predicate symbols to conceptual relations of the conceptualisation.

If is an ontological commitment for , then commits to by means of , while is the underlying conceptualisation of .

Ontology

Given a language with ontological commitment , an ontology for is a set of axioms designed in such a way that the set of its models approximates as best as possible the set of intended models of according to . An ontology commits to if:

1. has been designed to characterize .
2. approximates .

With this framework an ontology is a formal, explicit description of concepts in a domain of discourse. It is a set of logical axioms that accounts for the intended meaning of a formal vocabulary.

While Guarino provides a rigorous theoretical framework, Noy and Mcguinness (2001) introduce a complementary vocabulary for practical, scalable ontology development. They state that an ontology defines a common vocabulary to facilitate information exchange within a domain. It includes machine-interpretable definitions of basic domain concepts and their relations. This practical description relies on the following foundational elements:

• Concepts: These are the formal descriptions of the concepts in the domain of discourse. They are modelled in the ontology as classes.
• Properties: They describe various features and attributes of a concept; they are also called slots or roles.
• Facets: These are the role restrictions on the properties. The facet may apply these restrictions on either Value Type, Range of Values or Cardinality.
• Instances: The extensional realisations based on the intensional structures are called instances. For example, an ontology may define the class Car as an intensional concept that specifies the properties and structural characteristics that constitute cars. Individual vehicles, such as a particular Kia Sportage, correspond to instances that instantiate this class.
• Knowledge Base: The ontology, along with the instances of classes, forms a knowledge base.

STEP 1. Determine the domain and the scope of the ontology

In this step, we determine: (i) the domain covered by the ontology, (ii) the purpose of the ontology, (iii) the types of questions to which the information in the ontology should provide an answer, and (iv) the users and maintainers of the ontology. One way to determine the scope of the ontology is to use competency questions (Monfardini, Salamon, and Barcellos 2023). This is a set of questions that a knowledge base based on the ontology should be able to answer. Once defined, it can also be used to test the adequacy of a designed ontology.

STEP 2. Accessing existing ontologies for suitability

One should always consider using the existing ontology that can address the requirements. Only in the cases where no ontology exists should one be developed from scratch. If the existing ontology has addressable shortcomings, then it should be modified.

STEP 3. Enumerate important terms in the ontology

Initially, it is important to get a comprehensive list of terms without worrying about the overlap between the concepts they represent, relations among the terms or any properties of the concepts.

STEP 4. Define and organise classes hierarchically

The process of defining classes and organising the classes hierarchically during the ontology development can be executed using the following approaches:

• Top-Down Development: It starts with the definition of the most general concepts first and then subsequently describes more specialised concepts. For example, if the goal is to develop classes for an ontology of wine and food, one would start with general concepts of Wine and Food, then move on to specialisation. The class Wine would be specialised by first creating its subclasses: White Wine, Red wine and Rose Wine. We can further categorise the Red wine class, for example, into Syrah, Red Burgundy, Cabernet Sauvignon, and so on, thereby moving toward even more specialised concepts.
• Bottom-Up Development: This approach starts with defining the specific classes as the leaves, then groups them and defines a general class. It is methodologically inverse to Top-Down Development, i.e., we would first define Syrah, Red Burgundy, and Cabernet Sauvignon, and then group them under a common superclass Red Wine.
• Combination Development: This approach integrates elements of both the bottom-up and top-down approaches by identifying the most important concepts first—whether general or specific—and then either specialising or generalising as necessary. For instance, one might start by defining key concepts such as Cabernet Sauvignon and Wine, and then expand into subclasses or generalise them, grouping related concepts as appropriate to create a more versatile structure.

STEP 5. Assign properties to the class and define the facets

Once the classes are defined, the next step is to populate the slots or the properties of these classes. For the Wine and Food example the properties associated with the classes might represent color, body, flavour, sugar, content, etc. Once the properties are listed, they must be assigned to the relevant classes. There are two types of properties: (i) Data Property, which describes the intrinsic property of the class, and (ii) Object Property, which describes the relationship between two objects of different classes. The facets represent the specification of the values that can be used to populate the Data Properties. They can be one or all of the following:

• Cardinality: It defines the number of values a property can have. It may be described by minimum (N) and maximum (M) cardinality to describe the number of slot values more precisely. Minimum cardinality of N means that a slot must have at least N values. For example, the grape slot of a Wine has a minimum cardinality of 1: each wine is made of at least one variety of grape.
• Value Type: It describes the type of data type of values the properties can take.
• Domain & Range: Domain describes the classes that can use the data property, and range describes the restrictions in the property values.

Large Scale Ontologies with Multi-Decade Maintenance

One of the most extensively developed families of applied ontologies is found in the biomedical sciences. The Open Biological and Biomedical Ontology (OBO) Foundry (Smith et al. 2007) provides a coordinated suite of interoperable ontologies, of which the Gene Ontology (GO) (Ashburner et al. 2000) is among the most widely used, providing a controlled vocabulary for biological processes, molecular functions, and cellular components across organisms. These biomedical ontologies are notable for both their scale (GO alone contains over 40,000 terms) and their institutional maintenance infrastructure, features that are rare outside the life sciences and that few domain-specific ontologies can be expected to replicate.

In the financial and legal domains, ontologies have attracted growing interest as a means of formalising regulatory and contractual knowledge. The Financial Industry Business Ontology (FIBO), developed by the Enterprise Data Management Council and standardised through the Object Management Group, provides a modular OWL-based representation of financial instruments, legal entities, and market processes (Bennett 2013). FIBO has been adopted by major financial institutions as a data governance framework, enabling cross-system interoperability and regulatory reporting. In the legal domain, efforts such as LegalRuleML (Athan et al. 2015) and the Legal Knowledge Interchange Format have sought to represent normative reasoning and deontic logic within formal ontological structures, though full-scale deployment in live legal systems remains limited. These applications underscore a common motivation: in high-stakes, highly structured domains, informal or purely statistical representations of knowledge are insufficient, and ontological precision serves a governance function as much as a scientific one.

Semantic Web Ontologies

The Semantic Web tradition has produced a third influential strand of ontological application, centred on the OWL and RDF standards (Mcguinness et al. 2004) and realised at scale in projects such as DBpedia (Auer et al. 2007) and Wikidata. DBpedia, by extracting structured information from Wikipedia infoboxes and encoding it in RDF, made available a large, interlinked knowledge graph that has been used extensively in entity linking, question answering, and information retrieval research.

Natural Language Processing

In natural language processing and affective computing, ontological resources have played a similarly enabling role, though somewhat more modestly. SentiWordNet (Esuli, Sebastiani, et al. 2006) extended the WordNet lexical database with sentiment polarity scores, providing a structured resource for opinion mining that predated the widespread use of neural approaches. More recent work has sought to formalise opinion and affect at a structural level, moving beyond lexicons toward relational representations of the components of subjective language, a trajectory that informs the construction of the UOC ontology developed in this thesis.

It is important to acknowledge, however, that the utility of an ontology is not straightforwardly proportional to its scale. The large ontologies described above (GO, FIBO, DBpedia) are the products of decades of coordinated community effort, sustained institutional investment, and, in many cases, regulatory mandate. As opposed to these multi-decade domain-specific ontology infrastructures, the ontology developed in this thesis is a domain ontology of a modest scale. Its usefulness is intended for conceptual precision, which involves structural coherence within a targeted scope of opinion analysis.

Ontology in the Era of LLMs

This distinction has become particularly salient in the context of large language models and contemporary AI systems. LLMs, trained on vast corpora, have demonstrated remarkable capacity for implicit knowledge representation, raising the question of whether formal ontologies retain utility in an era of such models. The emerging consensus in the research literature suggests that they do, though the nature of that utility has shifted. Ontologies have been shown to improve LLM performance through structured prompting and retrieval-augmented generation, to provide interpretable scaffolding for knowledge graph construction and verification (Kommineni, König-Ries, and Samuel 2024), and to serve as evaluation frameworks against which the implicit knowledge of language models can be assessed (Alivanistos et al. 2022). Rather than being supplanted by neural approaches, formal ontological representations increasingly function as complementary structures providing the explicit, verifiable semantics that distributional models alone cannot guarantee.

In NLP tasks specifically, where the boundaries between sentiment, opinion, and emotion are conceptually contested, ontological grounding offers a principled means of disambiguating constructs that might otherwise be conflated in purely data-driven frameworks.

Weisfeiler-Lehman Isomorphism Test:

A test to investigate isomorphism in graphs.

Note: ### : injective function An injective function is also called a one-to-one function.

Definition: A function is injective if it maps distinct inputs to distinct outputs.

i.e. If , then

Cons:

• Highly discontinuous function.

• No guarantee to find the right function.

Primary Idea:

GNNs like Graph Convolution Network (GCN), Graph ATtention GAT, MPNN and GIN use only a single aggregation method with mean, sum and max aggregators being the most used. A limitation of this method is that aggregators fail to discriminate among messages when using a single GNN layer.

To address this, Corso et al. (2020) propose aggregators as continuous functions of multisets. These compute statistics—such as mean, max, or standard deviation—on neighboring nodes. The continuity of input spaces is emphasized: small input changes should yield small output changes for the aggregator.

They used the following aggregators: - Mean Aggregation : The most common message aggregator in the literature, wherein each node computes a weighted average or sum of its incoming messages. This can be calculated as:

Where $i$: target node, $d_i$: Degree of $i$ node, $j \in N(i)$: all the neighbours of $ i$, $X^{l}$: features vector of a neighbour node $j$ at current layer l.

• Maximum and Minimum Aggregations: ,

• Standard Deviation Aggregation: This is calculated as follows:

  

  Here, mean and standard deviation describe the neighborhood by measuring the average and spread of node features. Higher moments, such as skewness (3rd) and kurtosis (4th), can provide more detail about the neighborhood’s shape.

Degree-based scalers

They also use scalers as functions of the number of messages being aggregated (usually the node degree), which are multiplied by the aggregated value to amplify or attenuate the incoming messages.

where, is a constant value

-> Nodes in the neighbourhood, : Degree of incoming nodes of the neighbours.

, ,

Summation aggregation doesn’t generalise well to unseen graphs, as a small change in degree can amplify or attenuate the message and gradients exponentially. In PNA, using a logarithmic amplification is proposed to reduce this effect.

The PNA operator can then be realised in the framework of message passing, which includes edge features also:

: is the features of the edge , and are neural networks. : reduces the size of concatenated messages back to where is the dimension of the hidden features in the network.

import numpy as np
import torch
from torch_geometric.typing import Adj, OptTensor

import torch
from torch import Tensor
from torch.nn import ModuleList, Sequential, Linear, ReLU
from torch_geometric.nn.conv import MessagePassing
from torch_geometric.nn.inits import reset
from torch_geometric.utils import degree, add_self_loops

src_arr = torch.tensor(np.random.randint(0, 100, 10))
indexes = torch.tensor(np.random.randint(0, 3, 10))

print('index:', indexes)
print('src arr:', src_arr)

index: tensor([1, 0, 0, 1, 2, 0, 0, 0, 1, 1])
src arr: tensor([33, 74, 81, 91, 41, 45, 68, 96, 76, 78])

# we use torch_scatter as a mechanism to apply index-wise reduce operation.
reduced_output = scatter(src=src_arr, index=indexes, reduce="sum")
print(reduced_output)

tensor([364, 278,  41])

Degree Histogram

We require degree histogram when using PNA layer for calculating scalars.

sns.set_style('whitegrid')
fig, ax = plt.subplots()
x = np.arange(degree_hist.shape[0])
ax.bar(x, degree_hist)
ax.set_xlabel("Node Degree")
ax.set_ylabel("No. of nodes")
ax.set_title('Degree Histogram')
plt.show()

model = PNA_Network(degree_hist)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)
data = dataset[0]

start = time.process_time()
for epoch in range(1000):
    model.train()
    optimizer.zero_grad()
    out = model(data)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    
    if epoch%200 == 0:
        print(f"Time Elapsed: {time.process_time() - start}, Epoch: {epoch}, loss: {loss.item()}")
    loss.backward()
    optimizer.step()

Time Elapsed: 2.681476000000001, Epoch: 0, loss: 1.9427117109298706
Time Elapsed: 483.15815299999997, Epoch: 200, loss: 0.012473355047404766
Time Elapsed: 965.493997, Epoch: 400, loss: 0.011228213086724281
Time Elapsed: 1449.137884, Epoch: 600, loss: 0.06128396838903427
Time Elapsed: 1929.509851, Epoch: 800, loss: 0.001116628060117364

GIN Implementation

If all Aggregators except sum are removed and the central node is added using self loops. The PNA implementation can be reduced to an instance of GIN layer.

class GINConvSimple(MessagePassing):
    def __init__(self, in_channels: int, out_channels: int,                  
                 post_layers: int = 1, **kwargs):
        super(GINConvSimple, self).__init__(aggr=None, node_dim=0, **kwargs)
        self.in_channels = in_channels #no of input channels
        self.out_channels = out_channels #no

        # reducing aggregators to sum
        self.aggregators = [AGGREGATORS[aggr] for aggr in ['sum']]

      
        ## see what it does
        self.F_in = in_channels
        self.F_out = out_channels

      
        in_channels = (1 * 1) * self.F_in # 6 * 5 * in_channel #adjustment based on scalars
        
        ## this just stacks MLPs
        modules = [Linear(in_channels, self.F_out)] # just a linear layer
        for _ in range(post_layers-1):
            modules += [ReLU()]
            modules += [Linear(in_channels, self.F_out)]
        
        self.post_nn = Sequential(*modules)
        self.reset_parameters()

    def reset_parameters(self):
        reset(self.post_nn)

    def forward(self, x: Tensor, edge_index: Adj, edge_attr: OptTensor = None) -> Tensor:

        # propagate_type: (x: Tensor)
        edge_index, _ = add_self_loops(edge_index)
        out = self.propagate(edge_index, x=x, size=None)
        return self.post_nn(out)

    def message(self, x_j):
        return x_j

    def aggregate(self, inputs, index, dim_size = None):
        outs = [aggr(inputs, index, dim_size) for aggr in self.aggregators]
        
        out = torch.cat(outs, dim=-1)
        
        # deg = degree(index, dim_size, dtype=inputs.dtype).view(-1, 1)
        # outs = [sclr(out, deg, self.avg_degrees) for sclr in self.scalars]
        return out
    

class GIN_Network(torch.nn.Module):
    def __init__(self):
        
        super(GIN_Network, self).__init__()
        self.hid = 32
        self.pna_layer = GINConvSimple(dataset.num_features, dataset.num_classes)

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x = F.dropout(x, p=0.6, training=self.training)
        # print(f"Input shape Conv1: {x.shape}")
        x = self.pna_layer(x, edge_index)
        
        return F.log_softmax(x, dim=1)

model = GIN_Network()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)
data = dataset[0]

start = time.process_time()
for epoch in range(1000):
    model.train()
    optimizer.zero_grad()
    out = model(data)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    
    if epoch%200 == 0:
        print(f"Time Elapsed: {time.process_time() - start}, Epoch: {epoch}, loss: {loss.item()}")
    loss.backward()
    optimizer.step()

Time Elapsed: 0.2294729999998708, Epoch: 0, loss: 1.9651533365249634
Time Elapsed: 62.08905800000002, Epoch: 200, loss: 0.02035176381468773
Time Elapsed: 124.28675900000007, Epoch: 400, loss: 0.012289958074688911
Time Elapsed: 186.90790199999992, Epoch: 600, loss: 0.011557257734239101
Time Elapsed: 248.96050700000023, Epoch: 800, loss: 0.01037731021642685

Fourier Transform

It is a mathematical function , that decomposes a function into its constituent frequencies.

Transformation . It has the following property that makes it desirable to us:

, i.e.  is an Invertible function.

, i.e. It transforms convolution function into matrix multiplication function.

This implies

# implementation of u_{n} and the function to generate U i.e. eq ii
def matrix_U(N):
    # generates tensors for x in 0 to N-1 for eq (iii), 
    u = lambda n, N: np.cos(2 * np.pi/N * np.arange(N)) - 1j * np.sin(2 * np.pi/N * n * np.arange(N)) 
    U = np.empty((N, 0))
    for n in range(N):
        U = np.c_[U, u(n, N)] #stack each u as a column in U
    return U

# implementing 

def fourier_transform(v):
    N = v.shape[0]
    U = matrix_U(N)
    return U @ v

def inverse_fourier_transform(v):
    N = v.shape[0]
    U = matrix_U(N)
    return (U.conj().transpose() @ v) / N

Graph Laplacian

It is computed as:

where, : The Degree Matrix, : Adjacency Matrix

Laplacian is a measure of “smoothness”. When applied to a graph signal, it informs how much a node differs from the average of its neighbours. - Low Frequency: The signal changes slowly across the graph. (neighbors have very similar values) - High Frequency: The signal changes rapidly. (neighbors have very different values)

Convolution on a Graph

We using the graph’s “waves” (the matrix ), we can define convolution as:

The Major Problem: Calculating (eigenvectors) requires decomposing a huge matrix . For a graph with millions of nodes, this is computationally impossible.

dataset = Planetoid(root="tutorial1", name="Cora")
print(f"Cora Graphs, {len(dataset)}")
print(f"Classes: {dataset.num_classes}")
print(f"Features: {dataset.num_features}")

Cora Graphs, 1
Classes: 7
Features: 1433

## Model using GCNN 
class GCNN_Network(torch.nn.Module):
    def __init__(self):
        super(GCNN_Network, self).__init__()
        self.hid = 32
        self.conv1 = GCNConv(dataset.num_features, self.hid)
        self.conv2 = GCNConv(self.hid, dataset.num_classes)

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x = F.dropout(x, p=0.6, training=self.training)
        # print(f"Input shape Conv1: {x.shape}")
        x = self.conv1(x, edge_index)
        # print(f"Output shape Conv1, hidden->{self.hid} : output->{x.shape}")
        x = F.elu(x)
        x = F.dropout(x, p=0.6, training=self.training)
        x = self.conv2(x, edge_index)
        # print(f"Output shape Conv2, heads : {x.shape}")
        return F.log_softmax(x, dim=1)

model = GCNN_Network()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)
data = dataset[0]

for epoch in range(1000):
    model.train()
    optimizer.zero_grad()
    out = model(data)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    
    if epoch%200 == 0:
        print(loss)
    loss.backward()
    optimizer.step()

tensor(1.9614, grad_fn=<NllLossBackward0>)
tensor(0.0526, grad_fn=<NllLossBackward0>)
tensor(0.0276, grad_fn=<NllLossBackward0>)
tensor(0.0156, grad_fn=<NllLossBackward0>)
tensor(0.0157, grad_fn=<NllLossBackward0>)

## Model using GCNN 
class ChebConv_Network(torch.nn.Module):
    def __init__(self):
        super(ChebConv_Network, self).__init__()
        self.hid = 32
        self.conv1 = ChebConv(dataset.num_features, self.hid, K=2)
        self.conv2 = ChebConv(self.hid, dataset.num_classes, K=2)

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x = F.dropout(x, p=0.6, training=self.training)
        # print(f"Input shape Conv1: {x.shape}")
        x = self.conv1(x, edge_index)
        # print(f"Output shape Conv1, hidden->{self.hid} : output->{x.shape}")
        x = F.elu(x)
        x = F.dropout(x, p=0.6, training=self.training)
        x = self.conv2(x, edge_index)
        # print(f"Output shape Conv2, heads : {x.shape}")
        return F.log_softmax(x, dim=1)

model = ChebConv_Network()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)
data = dataset[0]

for epoch in range(1000):
    model.train()
    optimizer.zero_grad()
    out = model(data)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    
    if epoch%200 == 0:
        print(loss)
    loss.backward()
    optimizer.step()

tensor(2.1998, grad_fn=<NllLossBackward0>)
tensor(0.0090, grad_fn=<NllLossBackward0>)
tensor(0.0222, grad_fn=<NllLossBackward0>)
tensor(0.0101, grad_fn=<NllLossBackward0>)
tensor(0.0084, grad_fn=<NllLossBackward0>)

Implementing steps of GAT (simple)

STEPS:

1. Linear Transformation
2. Attention Calculation
3. Weighing input with attention weight

in_features = 5
out_features = 2
nb_nodes = 3

W = torch.nn.Parameter(torch.zeros(size=(in_features, out_features)))
torch.nn.init.xavier_uniform(W.data, gain=1.414)
inpx = torch.rand(nb_nodes, in_features)
h = torch.mm(inpx, W)

N = h.size()[0]

/var/folders/kr/021dhy4100546918_56lpp4m0000gn/T/ipykernel_14371/2968647810.py:6: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
  torch.nn.init.xavier_uniform(W.data, gain=1.414)

a = torch.nn.Parameter(torch.zeros(size=(2*out_features, 1)))
torch.nn.init.xavier_normal(a.data, gain=1.414)

leaky_relu = torch.nn.LeakyReLU(0.2)
a_input = torch.cat([h.repeat(1, N).view(N * N, -1), h.repeat(N, 1)], dim=1).view(N, -1, 2 * out_features)

e = leaky_relu(torch.matmul(a_input, a).squeeze(2))

/var/folders/kr/021dhy4100546918_56lpp4m0000gn/T/ipykernel_14371/732829903.py:2: UserWarning: nn.init.xavier_normal is now deprecated in favor of nn.init.xavier_normal_.
  torch.nn.init.xavier_normal(a.data, gain=1.414)

e.shape, inpx.shape

(torch.Size([3, 3]), torch.Size([3, 5]))

print(a_input.shape, a.shape)
print("")
print(torch.matmul(a_input, a).shape)
print("")

torch.Size([3, 3, 4]) torch.Size([4, 1])

torch.Size([3, 3, 1])

adj = torch.randint(2, (3, 3))
zero_vec  = -9e15*torch.ones_like(e)
e
attention = torch.where(adj>0, e, zero_vec)

attention = F.softmax(attention, dim=1)
h_prime = torch.matmul(attention, h)

h

tensor([[ 0.7161, -0.3945],
        [ 0.5494, -0.4554],
        [-0.1075, -0.5380]], grad_fn=<MmBackward0>)

h_prime

tensor([[ 0.2763, -0.4898],
        [-0.1075, -0.5380],
        [ 0.7161, -0.3945]], grad_fn=<MmBackward0>)

test_arr = torch.tensor([1,2,3])

Exponential Linear Unit (ELU)

By default,

GATLayer

Building GAT as a custorm pytorch Layer

There is no implementation of Multi-headed Attention (MHA) in this case. It is made for node classification problem.

class GATLayer(torch.nn.Module):
    def __init__(self, in_features, out_features, dropout, concat=True):
        super(GATLayer, self).__init__()
        self.n_in_features = in_features
        self.n_out_features = out_features
        
        self.W = torch.nn.Parameter(torch.zeros(size=(in_features, out_features)))
        torch.nn.init.xavier_uniform(self.W.data, gain=1.414)
        
        self.a = torch.nn.Parameter(torch.zeros(size=(2*out_features, 1)))
        torch.nn.init.xavier_normal(self.a.data, gain=1.414)
        self.leakyrelu = torch.nn.LeakyReLU(0.2)

    def forward(self, input_tensor, adj):
        assert input_tensor.shape[-1] == self.n_in_features, "number of features is different"
        h = torch.mm(input_tensor, self.W)
        N = h.size()[0]
        a_input = torch.cat([h.repeat(1, N).view(N * N, -1), h.repeat(N, 1)], dim=1).view(N, -1, 2 * out_features)
        e = self.leakyrelu(torch.matmul(a_input, a).squeeze(2))
        
        # adj = torch.randint(2, (3, 3))
        zero_vec  = -9e15*torch.ones_like(e)
        attention = torch.where(adj > 0, e, zero_vec)
        attention = F.softmax(attention, dim=1)
        attention = F.dropout(attention, self.dropout, training=self.training)
        h_prime   = torch.matmul(attention, h)
        if self.concat:
            return F.elu(h_prime)
        else:
            return h_prime

class GAT(torch.nn.Module):
    def __init__(self):
        super(GAT, self).__init__()

        self.hid = 8
        self.in_head = 8
        self.out_head = 1

        self.conv1 = GATConv(dataset.num_features, self.hid, heads=self.in_head, dropout=0.6)
        self.conv2 = GATConv(self.hid*self.in_head, dataset.num_classes, concat=False, heads=self.out_head, dropout=0.6)

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x = F.dropout(x, p=0.6, training=self.training)
        # print(f"Input shape Conv1: {x.shape}")
        x = self.conv1(x, edge_index)
        # print(f"Output shape Conv1, heads {self.in_head}: {x.shape}")
        x = F.elu(x)
        x = F.dropout(x, p=0.6, training=self.training)
        x = self.conv2(x, edge_index)
        # print(f"Output shape Conv2, heads {self.out_head}: {x.shape}")
        return F.log_softmax(x, dim=1)

data = dataset[0]

model = GAT()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)

for epoch in range(1000):
    model.train()
    optimizer.zero_grad()
    out = model(data)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    
    if epoch%200 == 0:
        print(loss)
    loss.backward()
    optimizer.step()

tensor(2.0154, grad_fn=<NllLossBackward0>)
tensor(0.4505, grad_fn=<NllLossBackward0>)
tensor(0.2827, grad_fn=<NllLossBackward0>)
tensor(0.3186, grad_fn=<NllLossBackward0>)
tensor(0.3924, grad_fn=<NllLossBackward0>)