commit
083f1b589b
27 changed files with 2400 additions and 0 deletions
@ -0,0 +1,661 @@ |
|||
GNU AFFERO GENERAL PUBLIC LICENSE |
|||
Version 3, 19 November 2007 |
|||
|
|||
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/> |
|||
Everyone is permitted to copy and distribute verbatim copies |
|||
of this license document, but changing it is not allowed. |
|||
|
|||
Preamble |
|||
|
|||
The GNU Affero General Public License is a free, copyleft license for |
|||
software and other kinds of works, specifically designed to ensure |
|||
cooperation with the community in the case of network server software. |
|||
|
|||
The licenses for most software and other practical works are designed |
|||
to take away your freedom to share and change the works. By contrast, |
|||
our General Public Licenses are intended to guarantee your freedom to |
|||
share and change all versions of a program--to make sure it remains free |
|||
software for all its users. |
|||
|
|||
When we speak of free software, we are referring to freedom, not |
|||
price. Our General Public Licenses are designed to make sure that you |
|||
have the freedom to distribute copies of free software (and charge for |
|||
them if you wish), that you receive source code or can get it if you |
|||
want it, that you can change the software or use pieces of it in new |
|||
free programs, and that you know you can do these things. |
|||
|
|||
Developers that use our General Public Licenses protect your rights |
|||
with two steps: (1) assert copyright on the software, and (2) offer |
|||
you this License which gives you legal permission to copy, distribute |
|||
and/or modify the software. |
|||
|
|||
A secondary benefit of defending all users' freedom is that |
|||
improvements made in alternate versions of the program, if they |
|||
receive widespread use, become available for other developers to |
|||
incorporate. Many developers of free software are heartened and |
|||
encouraged by the resulting cooperation. However, in the case of |
|||
software used on network servers, this result may fail to come about. |
|||
The GNU General Public License permits making a modified version and |
|||
letting the public access it on a server without ever releasing its |
|||
source code to the public. |
|||
|
|||
The GNU Affero General Public License is designed specifically to |
|||
ensure that, in such cases, the modified source code becomes available |
|||
to the community. It requires the operator of a network server to |
|||
provide the source code of the modified version running there to the |
|||
users of that server. Therefore, public use of a modified version, on |
|||
a publicly accessible server, gives the public access to the source |
|||
code of the modified version. |
|||
|
|||
An older license, called the Affero General Public License and |
|||
published by Affero, was designed to accomplish similar goals. This is |
|||
a different license, not a version of the Affero GPL, but Affero has |
|||
released a new version of the Affero GPL which permits relicensing under |
|||
this license. |
|||
|
|||
The precise terms and conditions for copying, distribution and |
|||
modification follow. |
|||
|
|||
TERMS AND CONDITIONS |
|||
|
|||
0. Definitions. |
|||
|
|||
"This License" refers to version 3 of the GNU Affero General Public License. |
|||
|
|||
"Copyright" also means copyright-like laws that apply to other kinds of |
|||
works, such as semiconductor masks. |
|||
|
|||
"The Program" refers to any copyrightable work licensed under this |
|||
License. Each licensee is addressed as "you". "Licensees" and |
|||
"recipients" may be individuals or organizations. |
|||
|
|||
To "modify" a work means to copy from or adapt all or part of the work |
|||
in a fashion requiring copyright permission, other than the making of an |
|||
exact copy. The resulting work is called a "modified version" of the |
|||
earlier work or a work "based on" the earlier work. |
|||
|
|||
A "covered work" means either the unmodified Program or a work based |
|||
on the Program. |
|||
|
|||
To "propagate" a work means to do anything with it that, without |
|||
permission, would make you directly or secondarily liable for |
|||
infringement under applicable copyright law, except executing it on a |
|||
computer or modifying a private copy. Propagation includes copying, |
|||
distribution (with or without modification), making available to the |
|||
public, and in some countries other activities as well. |
|||
|
|||
To "convey" a work means any kind of propagation that enables other |
|||
parties to make or receive copies. Mere interaction with a user through |
|||
a computer network, with no transfer of a copy, is not conveying. |
|||
|
|||
An interactive user interface displays "Appropriate Legal Notices" |
|||
to the extent that it includes a convenient and prominently visible |
|||
feature that (1) displays an appropriate copyright notice, and (2) |
|||
tells the user that there is no warranty for the work (except to the |
|||
extent that warranties are provided), that licensees may convey the |
|||
work under this License, and how to view a copy of this License. If |
|||
the interface presents a list of user commands or options, such as a |
|||
menu, a prominent item in the list meets this criterion. |
|||
|
|||
1. Source Code. |
|||
|
|||
The "source code" for a work means the preferred form of the work |
|||
for making modifications to it. "Object code" means any non-source |
|||
form of a work. |
|||
|
|||
A "Standard Interface" means an interface that either is an official |
|||
standard defined by a recognized standards body, or, in the case of |
|||
interfaces specified for a particular programming language, one that |
|||
is widely used among developers working in that language. |
|||
|
|||
The "System Libraries" of an executable work include anything, other |
|||
than the work as a whole, that (a) is included in the normal form of |
|||
packaging a Major Component, but which is not part of that Major |
|||
Component, and (b) serves only to enable use of the work with that |
|||
Major Component, or to implement a Standard Interface for which an |
|||
implementation is available to the public in source code form. A |
|||
"Major Component", in this context, means a major essential component |
|||
(kernel, window system, and so on) of the specific operating system |
|||
(if any) on which the executable work runs, or a compiler used to |
|||
produce the work, or an object code interpreter used to run it. |
|||
|
|||
The "Corresponding Source" for a work in object code form means all |
|||
the source code needed to generate, install, and (for an executable |
|||
work) run the object code and to modify the work, including scripts to |
|||
control those activities. However, it does not include the work's |
|||
System Libraries, or general-purpose tools or generally available free |
|||
programs which are used unmodified in performing those activities but |
|||
which are not part of the work. For example, Corresponding Source |
|||
includes interface definition files associated with source files for |
|||
the work, and the source code for shared libraries and dynamically |
|||
linked subprograms that the work is specifically designed to require, |
|||
such as by intimate data communication or control flow between those |
|||
subprograms and other parts of the work. |
|||
|
|||
The Corresponding Source need not include anything that users |
|||
can regenerate automatically from other parts of the Corresponding |
|||
Source. |
|||
|
|||
The Corresponding Source for a work in source code form is that |
|||
same work. |
|||
|
|||
2. Basic Permissions. |
|||
|
|||
All rights granted under this License are granted for the term of |
|||
copyright on the Program, and are irrevocable provided the stated |
|||
conditions are met. This License explicitly affirms your unlimited |
|||
permission to run the unmodified Program. The output from running a |
|||
covered work is covered by this License only if the output, given its |
|||
content, constitutes a covered work. This License acknowledges your |
|||
rights of fair use or other equivalent, as provided by copyright law. |
|||
|
|||
You may make, run and propagate covered works that you do not |
|||
convey, without conditions so long as your license otherwise remains |
|||
in force. You may convey covered works to others for the sole purpose |
|||
of having them make modifications exclusively for you, or provide you |
|||
with facilities for running those works, provided that you comply with |
|||
the terms of this License in conveying all material for which you do |
|||
not control copyright. Those thus making or running the covered works |
|||
for you must do so exclusively on your behalf, under your direction |
|||
and control, on terms that prohibit them from making any copies of |
|||
your copyrighted material outside their relationship with you. |
|||
|
|||
Conveying under any other circumstances is permitted solely under |
|||
the conditions stated below. Sublicensing is not allowed; section 10 |
|||
makes it unnecessary. |
|||
|
|||
3. Protecting Users' Legal Rights From Anti-Circumvention Law. |
|||
|
|||
No covered work shall be deemed part of an effective technological |
|||
measure under any applicable law fulfilling obligations under article |
|||
11 of the WIPO copyright treaty adopted on 20 December 1996, or |
|||
similar laws prohibiting or restricting circumvention of such |
|||
measures. |
|||
|
|||
When you convey a covered work, you waive any legal power to forbid |
|||
circumvention of technological measures to the extent such circumvention |
|||
is effected by exercising rights under this License with respect to |
|||
the covered work, and you disclaim any intention to limit operation or |
|||
modification of the work as a means of enforcing, against the work's |
|||
users, your or third parties' legal rights to forbid circumvention of |
|||
technological measures. |
|||
|
|||
4. Conveying Verbatim Copies. |
|||
|
|||
You may convey verbatim copies of the Program's source code as you |
|||
receive it, in any medium, provided that you conspicuously and |
|||
appropriately publish on each copy an appropriate copyright notice; |
|||
keep intact all notices stating that this License and any |
|||
non-permissive terms added in accord with section 7 apply to the code; |
|||
keep intact all notices of the absence of any warranty; and give all |
|||
recipients a copy of this License along with the Program. |
|||
|
|||
You may charge any price or no price for each copy that you convey, |
|||
and you may offer support or warranty protection for a fee. |
|||
|
|||
5. Conveying Modified Source Versions. |
|||
|
|||
You may convey a work based on the Program, or the modifications to |
|||
produce it from the Program, in the form of source code under the |
|||
terms of section 4, provided that you also meet all of these conditions: |
|||
|
|||
a) The work must carry prominent notices stating that you modified |
|||
it, and giving a relevant date. |
|||
|
|||
b) The work must carry prominent notices stating that it is |
|||
released under this License and any conditions added under section |
|||
7. This requirement modifies the requirement in section 4 to |
|||
"keep intact all notices". |
|||
|
|||
c) You must license the entire work, as a whole, under this |
|||
License to anyone who comes into possession of a copy. This |
|||
License will therefore apply, along with any applicable section 7 |
|||
additional terms, to the whole of the work, and all its parts, |
|||
regardless of how they are packaged. This License gives no |
|||
permission to license the work in any other way, but it does not |
|||
invalidate such permission if you have separately received it. |
|||
|
|||
d) If the work has interactive user interfaces, each must display |
|||
Appropriate Legal Notices; however, if the Program has interactive |
|||
interfaces that do not display Appropriate Legal Notices, your |
|||
work need not make them do so. |
|||
|
|||
A compilation of a covered work with other separate and independent |
|||
works, which are not by their nature extensions of the covered work, |
|||
and which are not combined with it such as to form a larger program, |
|||
in or on a volume of a storage or distribution medium, is called an |
|||
"aggregate" if the compilation and its resulting copyright are not |
|||
used to limit the access or legal rights of the compilation's users |
|||
beyond what the individual works permit. Inclusion of a covered work |
|||
in an aggregate does not cause this License to apply to the other |
|||
parts of the aggregate. |
|||
|
|||
6. Conveying Non-Source Forms. |
|||
|
|||
You may convey a covered work in object code form under the terms |
|||
of sections 4 and 5, provided that you also convey the |
|||
machine-readable Corresponding Source under the terms of this License, |
|||
in one of these ways: |
|||
|
|||
a) Convey the object code in, or embodied in, a physical product |
|||
(including a physical distribution medium), accompanied by the |
|||
Corresponding Source fixed on a durable physical medium |
|||
customarily used for software interchange. |
|||
|
|||
b) Convey the object code in, or embodied in, a physical product |
|||
(including a physical distribution medium), accompanied by a |
|||
written offer, valid for at least three years and valid for as |
|||
long as you offer spare parts or customer support for that product |
|||
model, to give anyone who possesses the object code either (1) a |
|||
copy of the Corresponding Source for all the software in the |
|||
product that is covered by this License, on a durable physical |
|||
medium customarily used for software interchange, for a price no |
|||
more than your reasonable cost of physically performing this |
|||
conveying of source, or (2) access to copy the |
|||
Corresponding Source from a network server at no charge. |
|||
|
|||
c) Convey individual copies of the object code with a copy of the |
|||
written offer to provide the Corresponding Source. This |
|||
alternative is allowed only occasionally and noncommercially, and |
|||
only if you received the object code with such an offer, in accord |
|||
with subsection 6b. |
|||
|
|||
d) Convey the object code by offering access from a designated |
|||
place (gratis or for a charge), and offer equivalent access to the |
|||
Corresponding Source in the same way through the same place at no |
|||
further charge. You need not require recipients to copy the |
|||
Corresponding Source along with the object code. If the place to |
|||
copy the object code is a network server, the Corresponding Source |
|||
may be on a different server (operated by you or a third party) |
|||
that supports equivalent copying facilities, provided you maintain |
|||
clear directions next to the object code saying where to find the |
|||
Corresponding Source. Regardless of what server hosts the |
|||
Corresponding Source, you remain obligated to ensure that it is |
|||
available for as long as needed to satisfy these requirements. |
|||
|
|||
e) Convey the object code using peer-to-peer transmission, provided |
|||
you inform other peers where the object code and Corresponding |
|||
Source of the work are being offered to the general public at no |
|||
charge under subsection 6d. |
|||
|
|||
A separable portion of the object code, whose source code is excluded |
|||
from the Corresponding Source as a System Library, need not be |
|||
included in conveying the object code work. |
|||
|
|||
A "User Product" is either (1) a "consumer product", which means any |
|||
tangible personal property which is normally used for personal, family, |
|||
or household purposes, or (2) anything designed or sold for incorporation |
|||
into a dwelling. In determining whether a product is a consumer product, |
|||
doubtful cases shall be resolved in favor of coverage. For a particular |
|||
product received by a particular user, "normally used" refers to a |
|||
typical or common use of that class of product, regardless of the status |
|||
of the particular user or of the way in which the particular user |
|||
actually uses, or expects or is expected to use, the product. A product |
|||
is a consumer product regardless of whether the product has substantial |
|||
commercial, industrial or non-consumer uses, unless such uses represent |
|||
the only significant mode of use of the product. |
|||
|
|||
"Installation Information" for a User Product means any methods, |
|||
procedures, authorization keys, or other information required to install |
|||
and execute modified versions of a covered work in that User Product from |
|||
a modified version of its Corresponding Source. The information must |
|||
suffice to ensure that the continued functioning of the modified object |
|||
code is in no case prevented or interfered with solely because |
|||
modification has been made. |
|||
|
|||
If you convey an object code work under this section in, or with, or |
|||
specifically for use in, a User Product, and the conveying occurs as |
|||
part of a transaction in which the right of possession and use of the |
|||
User Product is transferred to the recipient in perpetuity or for a |
|||
fixed term (regardless of how the transaction is characterized), the |
|||
Corresponding Source conveyed under this section must be accompanied |
|||
by the Installation Information. But this requirement does not apply |
|||
if neither you nor any third party retains the ability to install |
|||
modified object code on the User Product (for example, the work has |
|||
been installed in ROM). |
|||
|
|||
The requirement to provide Installation Information does not include a |
|||
requirement to continue to provide support service, warranty, or updates |
|||
for a work that has been modified or installed by the recipient, or for |
|||
the User Product in which it has been modified or installed. Access to a |
|||
network may be denied when the modification itself materially and |
|||
adversely affects the operation of the network or violates the rules and |
|||
protocols for communication across the network. |
|||
|
|||
Corresponding Source conveyed, and Installation Information provided, |
|||
in accord with this section must be in a format that is publicly |
|||
documented (and with an implementation available to the public in |
|||
source code form), and must require no special password or key for |
|||
unpacking, reading or copying. |
|||
|
|||
7. Additional Terms. |
|||
|
|||
"Additional permissions" are terms that supplement the terms of this |
|||
License by making exceptions from one or more of its conditions. |
|||
Additional permissions that are applicable to the entire Program shall |
|||
be treated as though they were included in this License, to the extent |
|||
that they are valid under applicable law. If additional permissions |
|||
apply only to part of the Program, that part may be used separately |
|||
under those permissions, but the entire Program remains governed by |
|||
this License without regard to the additional permissions. |
|||
|
|||
When you convey a copy of a covered work, you may at your option |
|||
remove any additional permissions from that copy, or from any part of |
|||
it. (Additional permissions may be written to require their own |
|||
removal in certain cases when you modify the work.) You may place |
|||
additional permissions on material, added by you to a covered work, |
|||
for which you have or can give appropriate copyright permission. |
|||
|
|||
Notwithstanding any other provision of this License, for material you |
|||
add to a covered work, you may (if authorized by the copyright holders of |
|||
that material) supplement the terms of this License with terms: |
|||
|
|||
a) Disclaiming warranty or limiting liability differently from the |
|||
terms of sections 15 and 16 of this License; or |
|||
|
|||
b) Requiring preservation of specified reasonable legal notices or |
|||
author attributions in that material or in the Appropriate Legal |
|||
Notices displayed by works containing it; or |
|||
|
|||
c) Prohibiting misrepresentation of the origin of that material, or |
|||
requiring that modified versions of such material be marked in |
|||
reasonable ways as different from the original version; or |
|||
|
|||
d) Limiting the use for publicity purposes of names of licensors or |
|||
authors of the material; or |
|||
|
|||
e) Declining to grant rights under trademark law for use of some |
|||
trade names, trademarks, or service marks; or |
|||
|
|||
f) Requiring indemnification of licensors and authors of that |
|||
material by anyone who conveys the material (or modified versions of |
|||
it) with contractual assumptions of liability to the recipient, for |
|||
any liability that these contractual assumptions directly impose on |
|||
those licensors and authors. |
|||
|
|||
All other non-permissive additional terms are considered "further |
|||
restrictions" within the meaning of section 10. If the Program as you |
|||
received it, or any part of it, contains a notice stating that it is |
|||
governed by this License along with a term that is a further |
|||
restriction, you may remove that term. If a license document contains |
|||
a further restriction but permits relicensing or conveying under this |
|||
License, you may add to a covered work material governed by the terms |
|||
of that license document, provided that the further restriction does |
|||
not survive such relicensing or conveying. |
|||
|
|||
If you add terms to a covered work in accord with this section, you |
|||
must place, in the relevant source files, a statement of the |
|||
additional terms that apply to those files, or a notice indicating |
|||
where to find the applicable terms. |
|||
|
|||
Additional terms, permissive or non-permissive, may be stated in the |
|||
form of a separately written license, or stated as exceptions; |
|||
the above requirements apply either way. |
|||
|
|||
8. Termination. |
|||
|
|||
You may not propagate or modify a covered work except as expressly |
|||
provided under this License. Any attempt otherwise to propagate or |
|||
modify it is void, and will automatically terminate your rights under |
|||
this License (including any patent licenses granted under the third |
|||
paragraph of section 11). |
|||
|
|||
However, if you cease all violation of this License, then your |
|||
license from a particular copyright holder is reinstated (a) |
|||
provisionally, unless and until the copyright holder explicitly and |
|||
finally terminates your license, and (b) permanently, if the copyright |
|||
holder fails to notify you of the violation by some reasonable means |
|||
prior to 60 days after the cessation. |
|||
|
|||
Moreover, your license from a particular copyright holder is |
|||
reinstated permanently if the copyright holder notifies you of the |
|||
violation by some reasonable means, this is the first time you have |
|||
received notice of violation of this License (for any work) from that |
|||
copyright holder, and you cure the violation prior to 30 days after |
|||
your receipt of the notice. |
|||
|
|||
Termination of your rights under this section does not terminate the |
|||
licenses of parties who have received copies or rights from you under |
|||
this License. If your rights have been terminated and not permanently |
|||
reinstated, you do not qualify to receive new licenses for the same |
|||
material under section 10. |
|||
|
|||
9. Acceptance Not Required for Having Copies. |
|||
|
|||
You are not required to accept this License in order to receive or |
|||
run a copy of the Program. Ancillary propagation of a covered work |
|||
occurring solely as a consequence of using peer-to-peer transmission |
|||
to receive a copy likewise does not require acceptance. However, |
|||
nothing other than this License grants you permission to propagate or |
|||
modify any covered work. These actions infringe copyright if you do |
|||
not accept this License. Therefore, by modifying or propagating a |
|||
covered work, you indicate your acceptance of this License to do so. |
|||
|
|||
10. Automatic Licensing of Downstream Recipients. |
|||
|
|||
Each time you convey a covered work, the recipient automatically |
|||
receives a license from the original licensors, to run, modify and |
|||
propagate that work, subject to this License. You are not responsible |
|||
for enforcing compliance by third parties with this License. |
|||
|
|||
An "entity transaction" is a transaction transferring control of an |
|||
organization, or substantially all assets of one, or subdividing an |
|||
organization, or merging organizations. If propagation of a covered |
|||
work results from an entity transaction, each party to that |
|||
transaction who receives a copy of the work also receives whatever |
|||
licenses to the work the party's predecessor in interest had or could |
|||
give under the previous paragraph, plus a right to possession of the |
|||
Corresponding Source of the work from the predecessor in interest, if |
|||
the predecessor has it or can get it with reasonable efforts. |
|||
|
|||
You may not impose any further restrictions on the exercise of the |
|||
rights granted or affirmed under this License. For example, you may |
|||
not impose a license fee, royalty, or other charge for exercise of |
|||
rights granted under this License, and you may not initiate litigation |
|||
(including a cross-claim or counterclaim in a lawsuit) alleging that |
|||
any patent claim is infringed by making, using, selling, offering for |
|||
sale, or importing the Program or any portion of it. |
|||
|
|||
11. Patents. |
|||
|
|||
A "contributor" is a copyright holder who authorizes use under this |
|||
License of the Program or a work on which the Program is based. The |
|||
work thus licensed is called the contributor's "contributor version". |
|||
|
|||
A contributor's "essential patent claims" are all patent claims |
|||
owned or controlled by the contributor, whether already acquired or |
|||
hereafter acquired, that would be infringed by some manner, permitted |
|||
by this License, of making, using, or selling its contributor version, |
|||
but do not include claims that would be infringed only as a |
|||
consequence of further modification of the contributor version. For |
|||
purposes of this definition, "control" includes the right to grant |
|||
patent sublicenses in a manner consistent with the requirements of |
|||
this License. |
|||
|
|||
Each contributor grants you a non-exclusive, worldwide, royalty-free |
|||
patent license under the contributor's essential patent claims, to |
|||
make, use, sell, offer for sale, import and otherwise run, modify and |
|||
propagate the contents of its contributor version. |
|||
|
|||
In the following three paragraphs, a "patent license" is any express |
|||
agreement or commitment, however denominated, not to enforce a patent |
|||
(such as an express permission to practice a patent or covenant not to |
|||
sue for patent infringement). To "grant" such a patent license to a |
|||
party means to make such an agreement or commitment not to enforce a |
|||
patent against the party. |
|||
|
|||
If you convey a covered work, knowingly relying on a patent license, |
|||
and the Corresponding Source of the work is not available for anyone |
|||
to copy, free of charge and under the terms of this License, through a |
|||
publicly available network server or other readily accessible means, |
|||
then you must either (1) cause the Corresponding Source to be so |
|||
available, or (2) arrange to deprive yourself of the benefit of the |
|||
patent license for this particular work, or (3) arrange, in a manner |
|||
consistent with the requirements of this License, to extend the patent |
|||
license to downstream recipients. "Knowingly relying" means you have |
|||
actual knowledge that, but for the patent license, your conveying the |
|||
covered work in a country, or your recipient's use of the covered work |
|||
in a country, would infringe one or more identifiable patents in that |
|||
country that you have reason to believe are valid. |
|||
|
|||
If, pursuant to or in connection with a single transaction or |
|||
arrangement, you convey, or propagate by procuring conveyance of, a |
|||
covered work, and grant a patent license to some of the parties |
|||
receiving the covered work authorizing them to use, propagate, modify |
|||
or convey a specific copy of the covered work, then the patent license |
|||
you grant is automatically extended to all recipients of the covered |
|||
work and works based on it. |
|||
|
|||
A patent license is "discriminatory" if it does not include within |
|||
the scope of its coverage, prohibits the exercise of, or is |
|||
conditioned on the non-exercise of one or more of the rights that are |
|||
specifically granted under this License. You may not convey a covered |
|||
work if you are a party to an arrangement with a third party that is |
|||
in the business of distributing software, under which you make payment |
|||
to the third party based on the extent of your activity of conveying |
|||
the work, and under which the third party grants, to any of the |
|||
parties who would receive the covered work from you, a discriminatory |
|||
patent license (a) in connection with copies of the covered work |
|||
conveyed by you (or copies made from those copies), or (b) primarily |
|||
for and in connection with specific products or compilations that |
|||
contain the covered work, unless you entered into that arrangement, |
|||
or that patent license was granted, prior to 28 March 2007. |
|||
|
|||
Nothing in this License shall be construed as excluding or limiting |
|||
any implied license or other defenses to infringement that may |
|||
otherwise be available to you under applicable patent law. |
|||
|
|||
12. No Surrender of Others' Freedom. |
|||
|
|||
If conditions are imposed on you (whether by court order, agreement or |
|||
otherwise) that contradict the conditions of this License, they do not |
|||
excuse you from the conditions of this License. If you cannot convey a |
|||
covered work so as to satisfy simultaneously your obligations under this |
|||
License and any other pertinent obligations, then as a consequence you may |
|||
not convey it at all. For example, if you agree to terms that obligate you |
|||
to collect a royalty for further conveying from those to whom you convey |
|||
the Program, the only way you could satisfy both those terms and this |
|||
License would be to refrain entirely from conveying the Program. |
|||
|
|||
13. Remote Network Interaction; Use with the GNU General Public License. |
|||
|
|||
Notwithstanding any other provision of this License, if you modify the |
|||
Program, your modified version must prominently offer all users |
|||
interacting with it remotely through a computer network (if your version |
|||
supports such interaction) an opportunity to receive the Corresponding |
|||
Source of your version by providing access to the Corresponding Source |
|||
from a network server at no charge, through some standard or customary |
|||
means of facilitating copying of software. This Corresponding Source |
|||
shall include the Corresponding Source for any work covered by version 3 |
|||
of the GNU General Public License that is incorporated pursuant to the |
|||
following paragraph. |
|||
|
|||
Notwithstanding any other provision of this License, you have |
|||
permission to link or combine any covered work with a work licensed |
|||
under version 3 of the GNU General Public License into a single |
|||
combined work, and to convey the resulting work. The terms of this |
|||
License will continue to apply to the part which is the covered work, |
|||
but the work with which it is combined will remain governed by version |
|||
3 of the GNU General Public License. |
|||
|
|||
14. Revised Versions of this License. |
|||
|
|||
The Free Software Foundation may publish revised and/or new versions of |
|||
the GNU Affero General Public License from time to time. Such new versions |
|||
will be similar in spirit to the present version, but may differ in detail to |
|||
address new problems or concerns. |
|||
|
|||
Each version is given a distinguishing version number. If the |
|||
Program specifies that a certain numbered version of the GNU Affero General |
|||
Public License "or any later version" applies to it, you have the |
|||
option of following the terms and conditions either of that numbered |
|||
version or of any later version published by the Free Software |
|||
Foundation. If the Program does not specify a version number of the |
|||
GNU Affero General Public License, you may choose any version ever published |
|||
by the Free Software Foundation. |
|||
|
|||
If the Program specifies that a proxy can decide which future |
|||
versions of the GNU Affero General Public License can be used, that proxy's |
|||
public statement of acceptance of a version permanently authorizes you |
|||
to choose that version for the Program. |
|||
|
|||
Later license versions may give you additional or different |
|||
permissions. However, no additional obligations are imposed on any |
|||
author or copyright holder as a result of your choosing to follow a |
|||
later version. |
|||
|
|||
15. Disclaimer of Warranty. |
|||
|
|||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY |
|||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT |
|||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY |
|||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, |
|||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR |
|||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM |
|||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF |
|||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION. |
|||
|
|||
16. Limitation of Liability. |
|||
|
|||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING |
|||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS |
|||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY |
|||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE |
|||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF |
|||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD |
|||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), |
|||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF |
|||
SUCH DAMAGES. |
|||
|
|||
17. Interpretation of Sections 15 and 16. |
|||
|
|||
If the disclaimer of warranty and limitation of liability provided |
|||
above cannot be given local legal effect according to their terms, |
|||
reviewing courts shall apply local law that most closely approximates |
|||
an absolute waiver of all civil liability in connection with the |
|||
Program, unless a warranty or assumption of liability accompanies a |
|||
copy of the Program in return for a fee. |
|||
|
|||
END OF TERMS AND CONDITIONS |
|||
|
|||
How to Apply These Terms to Your New Programs |
|||
|
|||
If you develop a new program, and you want it to be of the greatest |
|||
possible use to the public, the best way to achieve this is to make it |
|||
free software which everyone can redistribute and change under these terms. |
|||
|
|||
To do so, attach the following notices to the program. It is safest |
|||
to attach them to the start of each source file to most effectively |
|||
state the exclusion of warranty; and each file should have at least |
|||
the "copyright" line and a pointer to where the full notice is found. |
|||
|
|||
<one line to give the program's name and a brief idea of what it does.> |
|||
Copyright (C) <year> <name of author> |
|||
|
|||
This program is free software: you can redistribute it and/or modify |
|||
it under the terms of the GNU Affero General Public License as published by |
|||
the Free Software Foundation, either version 3 of the License, or |
|||
(at your option) any later version. |
|||
|
|||
This program is distributed in the hope that it will be useful, |
|||
but WITHOUT ANY WARRANTY; without even the implied warranty of |
|||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
|||
GNU Affero General Public License for more details. |
|||
|
|||
You should have received a copy of the GNU Affero General Public License |
|||
along with this program. If not, see <http://www.gnu.org/licenses/>. |
|||
|
|||
Also add information on how to contact you by electronic and paper mail. |
|||
|
|||
If your software can interact with users remotely through a computer |
|||
network, you should also make sure that it provides a way for users to |
|||
get its source. For example, if your program is a web application, its |
|||
interface could display a "Source" link that leads users to an archive |
|||
of the code. There are many ways you could offer source, and different |
|||
solutions will be better for different programs; see section 13 for the |
|||
specific requirements. |
|||
|
|||
You should also get your employer (if you work as a programmer) or school, |
|||
if any, to sign a "copyright disclaimer" for the program, if necessary. |
|||
For more information on this, and how to apply and follow the GNU AGPL, see |
|||
<http://www.gnu.org/licenses/>. |
|||
@ -0,0 +1,11 @@ |
|||
A pluggable search engine designed to find alternative copies of textbooks. |
|||
Allows you to plug in a python module that provides textbook data for your |
|||
institution, then indexes that data in elasticsearch and lets people search for |
|||
books. It will try to find alternatives through various means. At the moment it |
|||
only searches the Internet Archive, and Open Library, however the plan is to |
|||
refine it to find better matches. |
|||
|
|||
Code is available under the GNU Affero General Public License |
|||
This means you must make any changes you make available if you run it on your |
|||
own server. See [here](https://www.gnu.org/licenses/why-affero-gpl.html) for |
|||
more info. |
|||
@ -0,0 +1,34 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
from urllib import quote |
|||
from json import loads, dumps |
|||
|
|||
import requests as req |
|||
|
|||
searchUrl = "https://archive.org/advancedsearch.php?q={0}&fl%5B%5D=avg_rating&fl%5B%5D=description&fl%5B%5D=identifier&fl%5B%5D=type&sort%5B%5D=&sort%5B%5D=&sort%5B%5D=&rows=50&page=1&output=json&callback=callback&save=yes#raw" |
|||
|
|||
def searchIA(title, author): |
|||
""" |
|||
Do a search on The Internet Archive for a book |
|||
""" |
|||
print "running a search" |
|||
requrl = searchUrl.format(quote(title + " " + author)) |
|||
try: |
|||
results = loads(req.get(requrl).text[9:][0:-1]) |
|||
except ValueError: |
|||
return [] |
|||
|
|||
rownum = results["responseHeader"]["params"]["rows"] |
|||
if rownum < 1: |
|||
print "Couldn't find results for %s %s" % (title, author) |
|||
return [] |
|||
docs = results["response"]["docs"] |
|||
urls = [] |
|||
for result in results["response"]["docs"][0:3]: |
|||
urls.append("https://archive.org/details/%s" % result["identifier"]) |
|||
return urls |
|||
|
|||
|
|||
# Example, search for David Hume's Enquiry Concerning Human Understanding |
|||
#for url in searchIA("Hume", "Enquiry Concerning Human Understanding"): |
|||
#print url |
|||
@ -0,0 +1 @@ |
|||
{"course":{"properties":{"textbooks":{"properties":{"author":{"type":"string","index":"analyzed"},"price":{"type":"string","index":"analyzed"},"title":{"type":"string","index":"analyzed"}}},"sections":{"properties":{"time":{"type":"string","index":"analyzed"},"title":{"type":"string","index":"analyzed"},"loc":{"type":"string","index":"analyzed"},"prof":{"type":"string","index":"analyzed"},"sem":{"type":"string","index":"analyzed"},"day":{"type":"string","index":"analyzed"}}}}}} |
|||
@ -0,0 +1,19 @@ |
|||
#! /usr/bin/racket |
|||
#lang racket |
|||
(require "schemadsl.rkt") |
|||
|
|||
(displayln |
|||
(make-mapping |
|||
"course" |
|||
`(,(estruct "sections" |
|||
`(,(str "title") |
|||
,(str "time") |
|||
,(str "loc") |
|||
,(str "prof") |
|||
,(str "sem") |
|||
,(str "day"))) |
|||
|
|||
,(estruct "textbooks" |
|||
`(,(str "title") |
|||
,(str "author") |
|||
,(str "price")))))) |
|||
@ -0,0 +1,62 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
from sys import argv |
|||
from hashlib import sha1 |
|||
|
|||
def truncate(docid): |
|||
""" |
|||
Truncate a document id to 12 digits |
|||
The document ID should be based on a |
|||
hash of unique identifiers |
|||
""" |
|||
return int(str(docid)[0:12]) |
|||
|
|||
def createResource(textbookInfo, course, dept, coursecode, docid): |
|||
""" |
|||
Create a document associated with a course |
|||
This document contains any/all resources associated |
|||
with that course |
|||
|
|||
example, |
|||
{ |
|||
'books': [], |
|||
'dept': 'COLLAB', |
|||
'code': '2C03', |
|||
'sections': [ |
|||
{ |
|||
'prof': 'Lisa Pender', |
|||
'sem': '2015/09/08 - 2015/12/08', |
|||
'day': 'Mo' |
|||
}, |
|||
{ |
|||
'prof': 'Staff', |
|||
'sem': '2015/09/08 - 2015/12/08', |
|||
'day': 'Th' |
|||
} |
|||
], |
|||
'title': 'COLLAB 2C03 - Sociology I' |
|||
} |
|||
""" |
|||
textbooks = textbookInfo(dept.strip(), coursecode.strip()) |
|||
|
|||
# We truncate the id so we can have nicer looking URLs |
|||
# Since the id will be used to point to the resource page for that course |
|||
_id = str(truncate(docid)) |
|||
|
|||
fields = { |
|||
"_id" : _id, |
|||
"textbooks" : textbooks, |
|||
"coursetitle" : "%s %s" % (dept.strip(), coursecode.strip()), |
|||
"courseinfo" : course |
|||
#"Syllabus" : "blah" |
|||
} |
|||
try: |
|||
revisions = list(localdb.revisions(_id)) |
|||
if not revisions: |
|||
return localdb.save(fields) |
|||
else: |
|||
rev = dict(revisions[0])["_rev"] |
|||
fields["_rev"] = rev |
|||
return localdb.save(fields) |
|||
except ResourceConflict: |
|||
print "Resource for %s already exists, not creating a new one" % (docid) |
|||
@ -0,0 +1,14 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
# predictive data |
|||
# switch to elasticsearch's prediction |
|||
|
|||
|
|||
|
|||
import database |
|||
import predictions |
|||
|
|||
class GOASearch(object): |
|||
def __init__(self): |
|||
return self |
|||
|
|||
@ -0,0 +1,349 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
from sys import argv |
|||
from itertools import chain, islice, izip as zip |
|||
from re import search, sub |
|||
from functools import total_ordering |
|||
|
|||
from sylla import textbookInfo |
|||
from collections import MutableMapping |
|||
|
|||
import datetime as dt |
|||
import lxml.html as lxh |
|||
import requests |
|||
import sys |
|||
import copy |
|||
|
|||
fall = "2159" |
|||
spring_summer = "2165" |
|||
winter = "2161" |
|||
|
|||
# threading stuff |
|||
import Queue as q |
|||
import threading as thd |
|||
|
|||
baseurl = "https://applicants.mcmaster.ca/psp/prepprd/EMPLOYEE/PSFT_LS/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL" |
|||
|
|||
searchurl = "https://csprd.mcmaster.ca/psc/prcsprd/EMPLOYEE/PSFT_LS/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL" |
|||
|
|||
custom_headers = { |
|||
"User-Agent" : "Mozilla/5.0 (X11; Linux x86_64; rv:41.0) Gecko/20100101 Firefox/41.0", |
|||
"Content-Type" : "application/x-www-form-urlencoded; charset=UTF-8", |
|||
} |
|||
|
|||
courseCodes1 = "ICAJAX=1&ICNAVTYPEDROPDOWN=1&ICType=Panel&ICElementNum=0&ICStateNum={0}&ICAction=CLASS_SRCH_WRK2_SSR_PB_SUBJ_SRCH%240&ICXPos=0&ICYPos=0&ResponsetoDiffFrame=-1&TargetFrameName=None&FacetPath=None&ICFocus=&ICSaveWarningFilter=0&ICChanged=-1&ICResubmit=0&ICSID=5tq9x%2Fjt42mf62Sh5z%2BrjxT0gT15kiIyQ2cecCSmRB4%3D&ICActionPrompt=false&ICFind=&ICAddCount=&ICAPPCLSDATA=&CLASS_SRCH_WRK2_STRM$45$={1}" |
|||
|
|||
courseCodes2 = "ICAJAX=1&ICNAVTYPEDROPDOWN=1&ICType=Panel&ICElementNum=0&ICStateNum={0}&ICAction=SSR_CLSRCH_WRK2_SSR_ALPHANUM_{1}&ICXPos=0&ICYPos=0&ResponsetoDiffFrame=-1&TargetFrameName=None&FacetPath=None&ICFocus=&ICSaveWarningFilter=0&ICChanged=-1&ICResubmit=0&ICSID=vIUgl6ZXw045S07EPbQw4RDzv7NmKCDdJFdT4CTRQNM%3D&ICActionPrompt=false&ICFind=&ICAddCount=&ICAPPCLSDATA=&CLASS_SRCH_WRK2_STRM$45$={2}" |
|||
|
|||
payload2 = "ICAJAX=1&ICNAVTYPEDROPDOWN=1&ICType=Panel&ICElementNum=0&ICStateNum={0}&ICAction=%23ICSave&ICXPos=0&ICYPos=0&ResponsetoDiffFrame=-1&TargetFrameName=None&FacetPath=None&ICFocus=&ICSaveWarningFilter=0&ICChanged=-1&ICResubmit=0&ICSID=aWx3w6lJ6d2wZui6hwRVSEnzsPgCA3afYJEFBLLkxe4%3D&ICActionPrompt=false&ICFind=&ICAddCount=&ICAPPCLSDATA=&CLASS_SRCH_WRK2_STRM$45$={1}" |
|||
|
|||
payload = "ICAJAX=1&ICNAVTYPEDROPDOWN=1&ICType=Panel&ICElementNum=0&ICStateNum={0}&ICAction=CLASS_SRCH_WRK2_SSR_PB_CLASS_SRCH&ICXPos=0&ICYPos=0&ResponsetoDiffFrame=-1&TargetFrameName=None&FacetPath=None&ICFocus=&ICSaveWarningFilter=0&ICChanged=-1&ICResubmit=0&ICSID=aWx3w6lJ6d2wZui6hwRVSEnzsPgCA3afYJEFBLLkxe4%3D&ICActionPrompt=false&ICFind=&ICAddCount=&ICAPPCLSDATA=&SSR_CLSRCH_WRK_SUBJECT$75$$0={1}&CLASS_SRCH_WRK2_STRM$45$={2}" |
|||
|
|||
|
|||
year = dt.date.today().year |
|||
month = dt.date.today().month |
|||
|
|||
days = { |
|||
"Mo" : 0, |
|||
"Tu" : 1, |
|||
"We" : 2, |
|||
"Th" : 3, |
|||
"Fr" : 4, |
|||
"Sa" : 5, |
|||
"Su" : 6 |
|||
} |
|||
|
|||
day_descs = { |
|||
"Mo" : "Monday Mon Mo", |
|||
"Tu" : "Tuesday Tues Tu Tue", |
|||
"We" : "Wednesday Wed We", |
|||
"Th" : "Thursday Th Thurs", |
|||
"Fr" : "Friday Fr Fri", |
|||
"Sa" : "Saturday Sat Sa", |
|||
"Su" : "Sunday Su Sun", |
|||
"T" : "TBA" |
|||
} |
|||
|
|||
def timeparse(time): |
|||
""" |
|||
Parse the time into numbers |
|||
""" |
|||
if len(time) == 7: |
|||
hour = int(time[0:2]) |
|||
minutes = int(time[3:5]) |
|||
half = time[5:7] |
|||
else: |
|||
hour = int(time[0]) |
|||
minutes = int(time[2:4]) |
|||
half = time[4:6] |
|||
if half == "PM": |
|||
if hour < 12: |
|||
hour = hour + 12 |
|||
|
|||
return (str(hour), str(minutes), half) |
|||
|
|||
class Class(object): |
|||
def __init__(self, dept, title, sections): |
|||
self.title = title.encode("UTF-8") |
|||
self.sections = sections |
|||
self.dept = dept |
|||
|
|||
def __repr__(self): |
|||
return repr((self.title, self.sections)) |
|||
|
|||
def __iter__(self): |
|||
return iter((self.title, sec) for sec in self.sections) |
|||
|
|||
def hasCode(self): |
|||
splitted = self.title.strip().split(" ") |
|||
return ((len(splitted) >= 2) and |
|||
(splitted[0].upper() == splitted[0]) and |
|||
(splitted[1].upper() == splitted[1])) |
|||
|
|||
@property |
|||
def code(self): |
|||
if self.hasCode(): |
|||
return self.title.strip().split(" ")[1].strip() |
|||
return False |
|||
|
|||
@property |
|||
def books(self): |
|||
if self.dept and self.code: |
|||
return textbookInfo(self.dept, self.code, withPrices=True) |
|||
return False |
|||
|
|||
@total_ordering |
|||
class Section(dict): |
|||
def __init__(self, time, loc, prof, sem): |
|||
self.time = time.encode("UTF-8") |
|||
self.loc = loc.encode("UTF-8") |
|||
self.prof = prof.encode("UTF-8") |
|||
self.sem = sem.encode("UTF-8") |
|||
self._date = False |
|||
self._day = False |
|||
|
|||
@property |
|||
def date(self): |
|||
if self.time != "TBA": |
|||
day, start, _, end = self.time.split() |
|||
|
|||
if self._day: |
|||
assert len(self._day) == 2 |
|||
day = self._day |
|||
else: |
|||
day = [day[n:n+2] for n in xrange(0, len(day)-1, 2)] |
|||
|
|||
self._date = (day, timeparse(start), timeparse(end)) |
|||
|
|||
return self._date |
|||
|
|||
return self.time |
|||
|
|||
@property |
|||
def day(self): |
|||
return self.date[0] |
|||
|
|||
@property |
|||
def start(self): |
|||
return self.date[1][0] + self.date[1][1] |
|||
|
|||
def __repr__(self): |
|||
return (""" |
|||
Time = %s, Location = %s, Instructor = %s, Semester Running = %s |
|||
""" % (self.date, self.loc, self.prof, self.sem)) |
|||
def __gt__(self, x): |
|||
if isinstance(self.day, list): |
|||
raise NotImplementedError |
|||
|
|||
if (self.date == "TBA" or |
|||
x.date == "TBA"): |
|||
return False |
|||
|
|||
return ((days[self.day] > days[x.day]) or |
|||
((self.day == x.day) and |
|||
(self.start > x.start))) |
|||
|
|||
def __eq__(self, x): |
|||
return (x.date == self.date and |
|||
x.prof == self.prof and |
|||
x.loc == self.loc and |
|||
x.sem == self.sem) |
|||
|
|||
|
|||
def getStateNum(html): |
|||
""" |
|||
Get the state num from Mosaic |
|||
This is unique to each requester |
|||
""" |
|||
parsed = lxh.fromstring(html) |
|||
return parsed.xpath(".//input[@name=\"ICStateNum\"]")[0].value |
|||
|
|||
def parseSection(section): |
|||
cols = section.xpath(".//td") |
|||
assert len(cols) == 4 |
|||
time, loc, prof, sem = [col.text_content().encode("UTF-8").strip() for col in cols] |
|||
|
|||
classinfo = Section(time, loc, prof, sem) |
|||
return classinfo |
|||
|
|||
def getSectionInfo(table): |
|||
trs = table.xpath(".//tr") |
|||
for tr in trs: |
|||
if tr.xpath("@id") and search(r"SSR_CLSRCH", tr.xpath("@id")[0]): |
|||
yield parseSection(tr) |
|||
|
|||
def parseColumns(subject, html): |
|||
parsed = lxh.fromstring(html) |
|||
|
|||
classInfo = (list(getSectionInfo(table)) for table in |
|||
islice((table for table in parsed.xpath(".//table") |
|||
if table.xpath("@id") and |
|||
search(r"ICField[0-9]+\$scroll", table.xpath("@id")[0])), 1, sys.maxint)) |
|||
|
|||
classNames = ((subject, span.text_content().strip()) for span in parsed.xpath(".//span") |
|||
if span.xpath("@id") and |
|||
search(r"DERIVED_CLSRCH_DESCR", span.xpath("@id")[0])) |
|||
|
|||
return zip(classNames, classInfo) |
|||
|
|||
def getCodes(html): |
|||
parsed = lxh.fromstring(html) |
|||
|
|||
return (code.text_content().encode("UTF-8") for code in |
|||
parsed.xpath("//span") |
|||
if code.xpath("@id") and |
|||
search(r"SSR_CLSRCH_SUBJ_SUBJECT\$[0-9]+", code.xpath("@id")[0])) |
|||
|
|||
class MosReq(object): |
|||
def __init__(self, semester): |
|||
self.semester = semester |
|||
s = requests.Session() |
|||
resp = s.get(baseurl, allow_redirects=True, headers=custom_headers).content |
|||
|
|||
# Let the server set some cookies before doing the searching |
|||
cookies = {} |
|||
for key, val in s.cookies.iteritems(): |
|||
cookies[key] = val |
|||
self.cookies = cookies |
|||
self.statenum = False |
|||
self.codes_ = [] |
|||
|
|||
def getlist(self, subject): |
|||
sys.stderr.write("Getting " + subject + "\n") |
|||
first_req = requests.get(searchurl, cookies=self.cookies).content |
|||
# for some reason Mosaic wants us to request it twice, ?????????????????? |
|||
self.statenum = getStateNum(first_req) |
|||
first_req = requests.post(searchurl, |
|||
data=payload.format(self.statenum, subject, self.semester), |
|||
cookies=self.cookies, |
|||
allow_redirects=False, |
|||
headers=custom_headers).content |
|||
# we make a first request to get the ICStateNum in case it thinks there are too many results |
|||
try: |
|||
self.statenum = getStateNum(first_req) |
|||
except IndexError: |
|||
pass |
|||
if "Your search will return over" in first_req: |
|||
|
|||
return requests.post(searchurl, |
|||
data=payload2.format(self.statenum, self.semester), |
|||
cookies=self.cookies, |
|||
allow_redirects=False, |
|||
headers=custom_headers).content |
|||
else: |
|||
return first_req |
|||
|
|||
def classes(self, subject): |
|||
return list(parseColumns(subject, self.getlist(subject))) |
|||
|
|||
def getCodes(self, letter): |
|||
sys.stderr.write("Getting letter " + letter + "\n") |
|||
first_req = requests.get(searchurl, cookies=self.cookies).content |
|||
self.statenum = getStateNum(first_req) |
|||
|
|||
self.statenum = getStateNum(requests.post(searchurl, |
|||
data=courseCodes1.format(self.statenum, self.semester), |
|||
cookies=self.cookies, |
|||
headers=custom_headers).content) |
|||
|
|||
return getCodes(requests.post(searchurl, |
|||
data=courseCodes2.format(self.statenum, letter, self.semester), |
|||
cookies=self.cookies, |
|||
allow_redirects=False, |
|||
headers=custom_headers).content) |
|||
@property |
|||
def codes(self): |
|||
if not self.codes_: |
|||
self.codes_ = list(chain.from_iterable( |
|||
map((lambda l: |
|||
self.getCodes(chr(l))), |
|||
xrange(65, 91)))) |
|||
return self.codes_ |
|||
|
|||
def request(codes, lists, semester): |
|||
requester = MosReq(semester) |
|||
while not codes.empty(): |
|||
code = codes.get() |
|||
try: |
|||
lists.put(requester.classes(code)) |
|||
except: |
|||
codes.task_done() |
|||
return |
|||
codes.task_done() |
|||
|
|||
|
|||
class CourseInfo(object): |
|||
def __init__(self, threadcount, semester): |
|||
self._codes = False |
|||
self.threadcount = threadcount |
|||
self.semester = semester |
|||
|
|||
@property |
|||
def codes(self): |
|||
if not self._codes: |
|||
req = MosReq(self.semester) |
|||
self._codes = req.codes |
|||
return self._codes |
|||
|
|||
def classes(self): |
|||
qcodes = q.Queue() |
|||
for code in self.codes: |
|||
qcodes.put(code) |
|||
lists = q.Queue() |
|||
threads = [] |
|||
thread = None |
|||
for i in xrange(self.threadcount): |
|||
thread = thd.Thread(group=None, target=request, args=(qcodes, lists, self.semester)) |
|||
threads.append(thread) |
|||
thread.start() |
|||
qcodes.join() |
|||
for t in threads: |
|||
t.join() |
|||
|
|||
sections = [] |
|||
while not lists.empty(): |
|||
sections.append(lists.get()) |
|||
|
|||
for cl in chain.from_iterable(sections): |
|||
new_sections = [] |
|||
for sec in cl[1]: |
|||
if len(sec.day) > 1: |
|||
for day in sec.day: |
|||
new_sections.append(copy.deepcopy(sec)) |
|||
new_sections[-1]._day = day |
|||
else: |
|||
sec._day = sec.day[0] |
|||
new_sections.append(sec) |
|||
yield Class(cl[0][0], sub("\xa0+", "", cl[0][1]), sorted(new_sections)) |
|||
|
|||
def getCourses(semester, threadcount=10): |
|||
return CourseInfo(threadcount, semester).classes() |
|||
|
|||
def allCourses(): |
|||
return chain.from_iterable( |
|||
(getCourses(sem, threadcount=10) |
|||
for sem in (fall, winter, spring_summer))) |
|||
|
|||
#for course in allCourses(): |
|||
#sys.stdout.write("%s, %s, %s, %s\n" % (course.title, course.code, course.dept, course.books)) |
|||
#print course.sections |
|||
@ -0,0 +1,9 @@ |
|||
from oersearch import Search |
|||
from classes import getCourses |
|||
from sylla import getTextbooks |
|||
|
|||
mcmasterSearch = Search("McMaster") |
|||
|
|||
mcmasterSearch.setup(getCourses) |
|||
|
|||
mcmasterSearch.run() |
|||
@ -0,0 +1,117 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
from sys import argv |
|||
from itertools import chain, islice, izip_longest, izip as zip |
|||
from re import search, sub |
|||
from functools import total_ordering |
|||
from re import sub |
|||
|
|||
import datetime as dt |
|||
import lxml.html as lxh |
|||
import requests |
|||
|
|||
# Purpose of this module is to download and parse syllabi from various departments |
|||
# In order to be corellated with individual courses |
|||
|
|||
class Price(object): |
|||
def __init__(self, amnt, status): |
|||
self.dollars = float(amnt[1:]) |
|||
self.status = status |
|||
|
|||
def __repr__(self): |
|||
return "$%s %s" % (repr(self.dollars), self.status) |
|||
|
|||
|
|||
class Book(object): |
|||
def __init__(self, title, price): |
|||
self.title = title |
|||
self.price = price |
|||
|
|||
def __repr__(self): |
|||
return '["%s", "%s"]' % (self.title, repr(self.price)) |
|||
|
|||
|
|||
def grouper(n, iterable, fillvalue=None): |
|||
"grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx" |
|||
args = [iter(iterable)] * n |
|||
return izip_longest(fillvalue=fillvalue, *args) |
|||
|
|||
searchUrl = "https://campusstore.mcmaster.ca/cgi-mcm/ws/txsub.pl?wsDEPTG1=%s&wsDEPTDESC1=&wsCOURSEG1=%s&crit_cnt=1" |
|||
|
|||
def normalize(word): |
|||
if len(word) > 1: |
|||
return ("%s%s" % |
|||
(word[0].upper(), |
|||
"".join(word[1:]).lower())) |
|||
return word |
|||
|
|||
def parseAuthor(author): |
|||
split = author.split(" ") |
|||
if len(split) <= 1: |
|||
return author |
|||
lastname = split[0] |
|||
firstname = split[1] |
|||
return "%s %s" % (firstname, lastname) |
|||
|
|||
def normwords(phrase): |
|||
words = phrase.split(" ") |
|||
return " ".join(map(normalize, words)) |
|||
|
|||
def books(dept, code, withPrices): |
|||
""" |
|||
Snatch me up a book title or three |
|||
""" |
|||
req = searchUrl % (dept, code) |
|||
|
|||
html = requests.get(req).text |
|||
|
|||
parsed = lxh.fromstring(html) |
|||
|
|||
pricelist = prices(parsed) |
|||
|
|||
for div in parsed.xpath(".//div"): |
|||
if (div.attrib.has_key("id") and |
|||
"prodDesc" in div.attrib["id"]): |
|||
|
|||
textbook = div.text_content() |
|||
author = sub(r',', '', |
|||
"".join( |
|||
(div.getparent() |
|||
.xpath(".//span[@class='inline']") |
|||
[0].text_content() |
|||
.split(":")[1:])).strip()) |
|||
price = pricelist.pop() |
|||
if withPrices: |
|||
yield (normwords(textbook), normwords(author), repr(price)) |
|||
else: |
|||
yield (normwords(textbook), normwords(author)) |
|||
|
|||
def prices(html): |
|||
""" |
|||
Get the prices from a search result page |
|||
""" |
|||
ps = [ |
|||
p.getparent().text_content().split()[0] |
|||
for p in html.xpath("//p/input[@type='checkbox']") |
|||
] |
|||
|
|||
try: |
|||
amts, stats = zip(*list(reversed(list(grouper(2, ps))))) |
|||
return map(Price, amts, stats) |
|||
except ValueError: |
|||
return [] |
|||
|
|||
def textbookInfo(dept, code, withPrices=False): |
|||
""" |
|||
Return all the textbooks for a course |
|||
""" |
|||
return list(books(dept, code, withPrices)) |
|||
|
|||
def humanities(): |
|||
""" |
|||
Download humanities syllabi |
|||
""" |
|||
return [] |
|||
|
|||
# Example, getting the course info for Personality Theory (PSYCH = Department, 2B03 = Course code) |
|||
# print list(courseInfo("PSYCH", "2B03")) |
|||
@ -0,0 +1,24 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
from urllib import quote |
|||
from json import loads, dumps |
|||
|
|||
import requests as req |
|||
|
|||
#query = "https://openlibrary.org/query.json?type=/type/edition&title=%s&author=%s" |
|||
searchurl = 'http://openlibrary.org/search.json?author=%s&title=%s' |
|||
|
|||
def bookUrls(title, author): |
|||
print title, author |
|||
if ":" in title: |
|||
title = title.split(":")[0] |
|||
requrl = searchurl % (quote(author), quote(title)) |
|||
results = loads(req.get(requrl).text) |
|||
for result in results["docs"][0:2]: |
|||
if result.has_key("edition_key"): |
|||
yield "https://openlibrary.org/books/%s" % result["edition_key"][0] |
|||
|
|||
# 'http://openlibrary.org/query.json?type=/type/edition&title=The+Personality+Puzzle' |
|||
|
|||
#for book in bookUrls("Philosophy Of Physics", "Tim Maudlin"): |
|||
#print book |
|||
@ -0,0 +1,153 @@ |
|||
##! /usr/bin/python2 |
|||
from itertools import groupby, chain |
|||
from sys import stdout |
|||
from functools import partial |
|||
from json import dumps |
|||
|
|||
def gensymer(): |
|||
n = [0] |
|||
def inner(): |
|||
result = str(n[0]) |
|||
n[0] += 1 |
|||
return result |
|||
return inner |
|||
|
|||
gensym = gensymer() |
|||
|
|||
def printTrie(graph, prev, trie, weight): |
|||
new_node = str(gensym()) |
|||
graph.node(new_node, "%s" % trie.letter) |
|||
graph.edge(prev, new_node, label="%.2f" % weight) |
|||
if not trie.children: |
|||
return |
|||
for child, weight in zip(trie.children, trie.ws): |
|||
printTrie(graph, new_node, child, weight) |
|||
|
|||
|
|||
class Trie(object): |
|||
def __init__(self, letter, children, ws): |
|||
self.letter = letter |
|||
self.children = children |
|||
self.ws = ws |
|||
|
|||
def probweight(suffixes): |
|||
weights = [float(s["value"]) for s in suffixes] |
|||
s = float(sum(weights)) |
|||
ws = [w/s for w in weights] |
|||
return ws |
|||
|
|||
def buildtrie(trie, suffixes): |
|||
""" |
|||
Build a trie, also known as a prefix tree, of all the possible completions |
|||
""" |
|||
trie.children = [] |
|||
for letter, suffs in suffixes: |
|||
ped = partition(suffs) |
|||
if any(map(lambda p: p[0], ped)): |
|||
# check if there are any children |
|||
trie.children.append(buildtrie(Trie(letter, [], probweight(suffs)), partition(suffs))) |
|||
else: |
|||
# we've reached the end of this word so just include the final letter |
|||
# [1] = there is a probability of 1 of reaching this single leaf node, |
|||
# since it is the only possible completion here |
|||
trie.children.append(Trie(letter, [], [1])) |
|||
return trie |
|||
|
|||
|
|||
def keyf(x): |
|||
if not x["key"]: |
|||
return "" |
|||
return x["key"][0] |
|||
|
|||
def tails(words): |
|||
for word in words: |
|||
yield { |
|||
"key" : word["key"][1:], |
|||
"value" : word["value"] |
|||
} |
|||
|
|||
def partition(words): |
|||
""" |
|||
Partition the words into different prefixes based on the first character |
|||
""" |
|||
groups = [ |
|||
(g[0], list(tails(g[1]))) |
|||
for g in groupby( |
|||
sorted(words, key=keyf), |
|||
key=keyf) |
|||
] |
|||
return groups |
|||
|
|||
|
|||
def flatten_helper(letter, trie): |
|||
return ([letter + child.letter for |
|||
child in trie.children], trie.children) |
|||
|
|||
def flatten(trie): |
|||
if not trie.children: |
|||
return trie.letter |
|||
prefixes, suffixes = flatten_helper(trie.letter, trie) |
|||
return [flatten(Trie(p, s2.children, s2.ws)) for p, s2 in zip(prefixes, suffixes)] |
|||
|
|||
def flattenlist(xs): |
|||
locs = [] |
|||
for x in xs: |
|||
if not isinstance(x, list): |
|||
locs.append(x) |
|||
else: |
|||
locs.extend(flattenlist(x)) |
|||
return locs |
|||
|
|||
def matchc(trie, prefix): |
|||
c = None |
|||
if len(prefix) > 1: |
|||
c = prefix[0] |
|||
else: |
|||
c = prefix |
|||
return [ch for ch in trie.children if ch.letter == c] |
|||
|
|||
def match(trie, word): |
|||
if not word: |
|||
return [] |
|||
m = matchc(trie, word[0]) |
|||
if not m: |
|||
return [] |
|||
else: |
|||
return [m[0]] + match(m[0], word[1:]) |
|||
|
|||
def complete(trie, word): |
|||
m = match(trie, word) |
|||
if len(word) != len(m): |
|||
return False |
|||
completions = [word+x[1:] for x in flattenlist(flatten(m[-1]))] |
|||
if len(completions) > 10: |
|||
return dumps(completions[0:10]) |
|||
return dumps(completions) |
|||
|
|||
def sortTrie(trie): |
|||
""" |
|||
Sort the children of each node in descending order |
|||
of the probability that each child would be the completion |
|||
of whatever that word is |
|||
""" |
|||
if not trie.children: |
|||
return |
|||
sortedChilds = sorted(zip(trie.children, trie.ws), key=lambda x: x[1], reverse=True) |
|||
trie.children = [x[0] for x in sortedChilds] |
|||
trie.ws = [x[1] for x in sortedChilds] |
|||
for child in trie.children: |
|||
sortTrie(child) |
|||
|
|||
def toTrie(words): |
|||
for word in words: |
|||
word["key"] = word["key"].lower() |
|||
trie = buildtrie(Trie("", [], [1]), partition(words)) |
|||
trie.ws = [1]*len(trie.children) |
|||
sortTrie(trie) |
|||
return trie |
|||
|
|||
def testkey(w): |
|||
return { |
|||
"key" : w, |
|||
"value" : "1" |
|||
} |
|||
@ -0,0 +1,67 @@ |
|||
#lang racket |
|||
|
|||
(require json) |
|||
|
|||
(define (root name type) |
|||
`(,name type)) |
|||
|
|||
(define ((prop |
|||
type |
|||
[index "analyzed"]) |
|||
name) |
|||
(define prop-hash (make-hash)) |
|||
(hash-set! prop-hash (string->symbol name) |
|||
`#hash( |
|||
(type . ,type) |
|||
(index . ,index))) |
|||
prop-hash) |
|||
|
|||
(define ((dict name) ps) |
|||
(let ([prop-vals (make-hash)] |
|||
[props-hsh (make-hash)]) |
|||
(for ([p ps]) |
|||
(hash-set! prop-vals |
|||
(car p) |
|||
(cdr p))) |
|||
(hash-set! props-hsh name prop-vals) |
|||
props-hsh)) |
|||
|
|||
(define (props ps) |
|||
(define props-hash (make-hash)) |
|||
(for ([p ps]) |
|||
(match p |
|||
[(hash-table (k v)) |
|||
(hash-set! props-hash k v)])) |
|||
`#hash((properties . ,props-hash))) |
|||
|
|||
(define str (prop "string")) |
|||
|
|||
(define num (prop "integer")) |
|||
|
|||
(define date (prop "date")) |
|||
|
|||
(define bool (prop "boolean" "not_analyzed")) |
|||
|
|||
(define (dictprop name d) |
|||
(define dict (make-hash)) |
|||
(hash-set! dict name d) |
|||
dict) |
|||
|
|||
(define (estruct name pairs) |
|||
(define estr (make-hash)) |
|||
(hash-set! estr |
|||
(string->symbol name) |
|||
(props pairs)) |
|||
estr) |
|||
|
|||
(define (make-mapping |
|||
type |
|||
decl) |
|||
(define mapping (make-hash)) |
|||
(hash-set! mapping |
|||
(string->symbol type) |
|||
(props decl)) |
|||
(jsexpr->string mapping)) |
|||
|
|||
(provide |
|||
(all-defined-out)) |
|||
@ -0,0 +1,35 @@ |
|||
<book> |
|||
<div class="text-clip toast" if={ opts.title && opts.author }> |
|||
<p> |
|||
<dt class="book-title text-center"> |
|||
<button onclick={ makeResourceGetter(this) } class="btn btn-link"> |
|||
{ opts.title } |
|||
</button> |
|||
</dt> |
|||
<dd> |
|||
<div if={ loading } class="loading"> |
|||
</div> |
|||
<p if={ iarchive }> |
|||
<a target="_blank" href="{ iarchive }"> |
|||
<button class="centered btn btn-link"> |
|||
Internet Archive Result |
|||
</button> |
|||
</a> |
|||
</p> |
|||
<p if={ openlib }> |
|||
<a target="_blank" href="{ openlib }"> |
|||
<button class="centered btn btn-link"> |
|||
Open Library Result |
|||
</button> |
|||
</a> |
|||
</p> |
|||
<p class="centered" if={ noResources }> |
|||
Couldn't find anything, sorry :( |
|||
</p> |
|||
</dd> |
|||
</p> |
|||
</div> |
|||
this.iarchive = false; |
|||
this.openlib = false; |
|||
this.noResources = false; |
|||
</book> |
|||
@ -0,0 +1,32 @@ |
|||
<class> |
|||
<div class="card-header"> |
|||
<div id='title'> { dept } { title } </div> |
|||
<div id='prof'> { prof } </div> |
|||
<div id='sem'> { sem } </div> |
|||
</div> |
|||
<div if={ books } class="card-body"> |
|||
<button onclick={ makeShow(this) } class="btn btn-primary show-button"> |
|||
<strong>Show Textbooks</strong> |
|||
</button> |
|||
<div if={ showBooks }> |
|||
<dl> |
|||
<book each={ books } |
|||
data="{ this }" |
|||
resources="" |
|||
title={ booktitle } |
|||
author={ bookauthor } |
|||
price={ bookprice }> |
|||
</book> |
|||
</dl> |
|||
</div> |
|||
</div> |
|||
<div class="toast" if={ !books }> |
|||
<p>No books at this time</p> |
|||
<p>Check back later, or verify the course has books</p> |
|||
</div> |
|||
</class> |
|||
|
|||
<script> |
|||
this.showBooks = false; |
|||
this.update(); |
|||
</script> |
|||
@ -0,0 +1,13 @@ |
|||
<results> |
|||
<div class="courses container"> |
|||
<row class="columns" each={ rows } data="{ this }" classrow={ row }></row> |
|||
</div> |
|||
this.rows = []; |
|||
var self = this; |
|||
results_passer.on("new_results", |
|||
function(data) { |
|||
console.log("new search results detected"); |
|||
self.rows = data; |
|||
self.update(); |
|||
}); |
|||
</results> |
|||
@ -0,0 +1,6 @@ |
|||
<row> |
|||
<class class="course text-ellipsis text-justify rounded card column col-md-3" each="{ classrow }" data="{ this }"></class> |
|||
|
|||
this.classrow = opts.classrow |
|||
</row> |
|||
|
|||
@ -0,0 +1,96 @@ |
|||
function makeResourceGetter(self) { |
|||
function getResources(ev) { |
|||
ev.preventDefault(); |
|||
self.loading = true; |
|||
self.update(); |
|||
var params = { |
|||
"title" : this.booktitle, |
|||
"author" : this.bookauthor |
|||
}; |
|||
var url = "http://localhost:8001/resources"; |
|||
console.log(params); |
|||
$.getJSON(url, { |
|||
data : JSON.stringify(params) |
|||
}).done(function(results) { |
|||
|
|||
if (results.iarchive) { |
|||
self.iarchive = results.iarchive[0]; |
|||
} |
|||
|
|||
if (results.openlib) { |
|||
self.openlib = results.openlib[0]; |
|||
} |
|||
|
|||
if (!(results.openlib && results.iarchive)) { |
|||
self.noResources = true; |
|||
} |
|||
self.loading = false; |
|||
self.update(); |
|||
}); |
|||
} |
|||
return getResources; |
|||
} |
|||
|
|||
function makeShow(self) { |
|||
return function() { |
|||
if (!self.showBooks) { |
|||
self.showBooks = true; |
|||
} |
|||
else { |
|||
self.showBooks = false; |
|||
} |
|||
self.update(); |
|||
}; |
|||
} |
|||
|
|||
function ResultsPasser() { |
|||
riot.observable(this); |
|||
return this; |
|||
} |
|||
|
|||
var results_passer = new ResultsPasser(); |
|||
|
|||
riot.mount("search"); |
|||
riot.mount("results"); |
|||
|
|||
function autocomplete(element, endpoint) { |
|||
// The element should be an input class
|
|||
$(element).autocomplete({ |
|||
source : endpoint, |
|||
my : "right top", |
|||
at : "left bottom", |
|||
collision : "none", |
|||
autofocus : true, |
|||
delay : 100 |
|||
}); |
|||
} |
|||
|
|||
function filterCourses(courses) { |
|||
return courses.filter( |
|||
function (c) { |
|||
return c.prof != "Staff"; |
|||
}); |
|||
} |
|||
|
|||
function take(n, xs) { |
|||
return xs.slice(0, n); |
|||
} |
|||
|
|||
function drop(n, xs) { |
|||
return xs.slice(n, xs.length); |
|||
} |
|||
|
|||
function groupsof(n, xs) { |
|||
var groups = []; |
|||
while (xs.length != 0) { |
|||
if (xs.length < n) { |
|||
groups.push({"row" : take(xs.length, xs)} ); |
|||
xs = drop(xs.length, xs); |
|||
} |
|||
else { |
|||
groups.push({"row" : take(n, xs)}); |
|||
xs = drop(n, xs); |
|||
} |
|||
} |
|||
return groups; |
|||
} |
|||
@ -0,0 +1,30 @@ |
|||
<search> |
|||
<form class="form-horizontal search-form" onsubmit={ submit } type="submit"method="get"> |
|||
<div class="form-group"> |
|||
<div class="col-sm-8 form-item"> |
|||
<input class="form-input" placeholder="Description" type="text" name="title"/> |
|||
</div> |
|||
<div class="col-sm-2 form-item"> |
|||
<select class="form-select" aria-labelledby="dLabel" name="sem"> |
|||
<option value="Fall">Fall</option> |
|||
<option value="Winter" selected>Winter</option> |
|||
<option value="Summer">Summer</option> |
|||
</select> |
|||
</div> |
|||
<div class="col-sm-2 form-item"> |
|||
<button class="btn btn-primary" type="submit">Search</button> |
|||
</div> |
|||
</div> |
|||
</form> |
|||
</search> |
|||
|
|||
function submit(ev) { |
|||
console.log("submitted"); |
|||
var params = $(ev.currentTarget).serialize(); |
|||
$.getJSON("http://localhost:8001/fc?"+params, |
|||
function(courses) { |
|||
var fcourses = filterCourses(courses); |
|||
var cgroups = groupsof(4, fcourses); |
|||
results_passer.trigger("new_results", cgroups); |
|||
}); |
|||
} |
|||
@ -0,0 +1,237 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
import elasticsearch |
|||
|
|||
from elasticsearch_dsl import FacetedSearch, Search, Q |
|||
from elasticsearch_dsl.aggs import Terms, DateHistogram |
|||
from sys import exit, stderr |
|||
from json import dumps, loads |
|||
from itertools import chain, imap |
|||
|
|||
from hashlib import sha1 |
|||
|
|||
from textbookExceptions import UnIndexable |
|||
|
|||
from mcmaster.classes import allCourses |
|||
|
|||
# Generic instance of elasticsearch right now |
|||
es = elasticsearch.Elasticsearch() |
|||
|
|||
def summarize(text): |
|||
splitted = text.split(" ") |
|||
if len(splitted) > 4: |
|||
return " ".join(splitted[0:4]) + ".." |
|||
return text |
|||
|
|||
def sectionToJSON(section): |
|||
return { |
|||
"prof" : section.prof, |
|||
"sem" : section.sem, |
|||
"day" : section.day |
|||
} |
|||
|
|||
def classToJSON(clss): |
|||
return { |
|||
"title" : clss.title, |
|||
"sections" : map(sectionToJSON, clss.sections), |
|||
"dept" : clss.dept, |
|||
"code" : clss.code, |
|||
"books" : list(clss.books) if clss.books else [] |
|||
} |
|||
|
|||
|
|||
def truncate(docid): |
|||
""" |
|||
Truncate a document id to 12 digits |
|||
The document ID should be based on a |
|||
hash of unique identifiers |
|||
""" |
|||
return int(str(docid)[0:12]) |
|||
|
|||
def hashsec(course): |
|||
""" |
|||
Hash a course into a usable id |
|||
""" |
|||
if not course["code"]: |
|||
code = "" |
|||
else: |
|||
code = course["code"] |
|||
if not course["title"]: |
|||
title = "" |
|||
else: |
|||
title = course["title"] |
|||
|
|||
if not course["sections"] or len(course["sections"]) < 1: |
|||
course["sections"][0] = "" |
|||
|
|||
if not (code or title): |
|||
raise UnIndexable(course) |
|||
|
|||
h = sha1() |
|||
h.update(code + title + course["sections"][0]["sem"]) |
|||
return int(h.hexdigest(), 16) |
|||
|
|||
def createIndex(name): |
|||
""" |
|||
This creates a new index in elasticsearch |
|||
An index is like a schema in a regular database |
|||
Create an elasticsearch index |
|||
|
|||
""" |
|||
indices = elasticsearch.client.IndicesClient(es) |
|||
|
|||
print indices.create(name) |
|||
with open("./course.json", "r") as mapping: |
|||
print indices.put_mapping("course", loads(mapping.read()), name) |
|||
|
|||
def indexListing(course): |
|||
""" |
|||
Index a specific course in the database (using the courses index) |
|||
example, |
|||
{ |
|||
'books': [], |
|||
'dept': 'COLLAB', |
|||
'code': '2C03', |
|||
'sections': [ |
|||
{ |
|||
'prof': 'Lisa Pender', |
|||
'sem': '2015/09/08 - 2015/12/08', |
|||
'day': 'Mo' |
|||
}, |
|||
{ |
|||
'prof': 'Staff', |
|||
'sem': '2015/09/08 - 2015/12/08', |
|||
'day': 'Th' |
|||
} |
|||
], |
|||
'title': 'COLLAB 2C03 - Sociology I' |
|||
} |
|||
|
|||
""" |
|||
courseID = hashsec(course) |
|||
print es.index(index="oersearch", |
|||
doc_type="course", |
|||
id=courseID, |
|||
body=course) |
|||
|
|||
# For every course we index, we also create a resource for it |
|||
# This should be an idempotent operation because we're putting it in couchdb |
|||
# And we're using the id obtained from the hash function, so it should just update the document |
|||
# no need to delete anything |
|||
#try: |
|||
#courseDept = course[0]["title"].strip().split(" ")[0].strip() |
|||
#courseCode = course[0]["title"].strip().split(" ")[1].strip() |
|||
#print "DEPARTMENT = \"%s\", COURSECODE = \"%s\"" % (courseDept, courseCode) |
|||
#print createResource(textbookInfo, course[0], courseDept, courseCode, courseID) |
|||
#except: |
|||
#print "Couldn't create the resource associated with %s" % course |
|||
|
|||
def termSearch(field): |
|||
""" |
|||
Make a term search (exact match) |
|||
""" |
|||
def t(term): |
|||
q = Q("term", |
|||
**{ |
|||
"sections."+field : term |
|||
}) |
|||
return q |
|||
return t |
|||
|
|||
def search(field): |
|||
""" |
|||
Make a match search |
|||
""" |
|||
def s(term): |
|||
q = Q("match", |
|||
**{ |
|||
field : term |
|||
}) |
|||
return q |
|||
return s |
|||
|
|||
def join(x, y): |
|||
""" |
|||
Join two queries |
|||
""" |
|||
return x & y |
|||
|
|||
def filterSections(secs): |
|||
""" |
|||
Get rid of tutorial sections |
|||
because they almost always have "Staff" as the instructor |
|||
This is just a heuristic of course |
|||
""" |
|||
filtered = [s for s in secs.sections if "Staff" not in s.prof] |
|||
if len(filtered) > 0: |
|||
return filtered |
|||
return False |
|||
|
|||
def searchTerms(terms): |
|||
""" |
|||
Run a search for courses |
|||
""" |
|||
|
|||
# A list of all the queries we want to run |
|||
qs = [searchers[field](term) for |
|||
field, term in |
|||
terms.iteritems() if |
|||
term and searchers.has_key(field)] |
|||
|
|||
if not qs: |
|||
# No queries = no results |
|||
return dumps([]) |
|||
|
|||
# Reduce joins all of the queries into one query |
|||
# It will search for the conjunction of all of them |
|||
# So that means it cares about each query equally |
|||
q = reduce(join, qs) |
|||
|
|||
s = (Search(using=es, index="oersearch") |
|||
.query(q))[0:100] # only return up to 100 results for now |
|||
|
|||
results = s.execute() |
|||
|
|||
filtered = [ |
|||
(secs, filterSections(secs)[0].to_dict()) # get rid of tutorials |
|||
for secs in results |
|||
if filterSections(secs) |
|||
] |
|||
results = [] |
|||
for obj, secs in filtered: |
|||
# Add the truncated course id |
|||
# This is used to point to the resource page for that course |
|||
secs["id"] = truncate(obj.meta.id) |
|||
secs["title"] = obj.title |
|||
if obj["dept"] not in secs["title"]: |
|||
secs["dept"] = obj.dept |
|||
if obj.books: |
|||
secs["books"] = [ |
|||
{ |
|||
"booktitle" : summarize(book[0].encode("ASCII")), |
|||
"bookauthor" : book[1].encode("ASCII"), |
|||
"bookprice" : book[2].encode("ASCII") |
|||
} |
|||
for book in obj.books |
|||
] |
|||
else: |
|||
secs["books"] = "" |
|||
results.append(secs) |
|||
|
|||
return dumps(results) |
|||
|
|||
|
|||
searchers = { |
|||
"title" : search("title"), |
|||
"loc" : search("loc"), |
|||
"time" : search("time"), |
|||
"prof" : search("prof"), |
|||
"day" : search("day"), |
|||
} |
|||
|
|||
#print searchTerms({"title" : "PHILOS"}) |
|||
|
|||
#for c in imap(classToJSON, allCourses()): |
|||
#try: |
|||
#print indexListing(c) |
|||
#except UnIndexable as e: |
|||
@ -0,0 +1,113 @@ |
|||
header { |
|||
color: #1c75bc; |
|||
} |
|||
|
|||
.body { |
|||
color: #1c75bc; |
|||
} |
|||
|
|||
a { |
|||
color: #1c75bc !important; |
|||
} |
|||
|
|||
.btn-primary { |
|||
background-color: #1c75bc !important; |
|||
} |
|||
|
|||
.courses { |
|||
margin-top: 100px; |
|||
max-width: 85%; |
|||
} |
|||
|
|||
.course { |
|||
margin-left: 5px !important; |
|||
margin-right: 5px !important; |
|||
margin-top: 5px !important; |
|||
margin-bottom: 5px !important; |
|||
} |
|||
|
|||
.search-form { |
|||
margin-top: 5%; |
|||
-webkit-appearance: none !important; |
|||
} |
|||
|
|||
.book-title { |
|||
margin-right: 10px !important; |
|||
} |
|||
|
|||
.form-item { |
|||
margin-left: 15px; |
|||
margin-right 15px; |
|||
} |
|||
|
|||
#title { |
|||
font-weight: bolder; |
|||
} |
|||
|
|||
.ui-autocomplete { |
|||
position: absolute; |
|||
top: 100%; |
|||
left: 0; |
|||
z-index: 1000; |
|||
float: left; |
|||
display: none; |
|||
min-width: 160px; |
|||
padding: 4px 0; |
|||
margin: 0 0 10px 25px; |
|||
list-style: none; |
|||
background-color: #ffffff; |
|||
border-color: #ccc; |
|||
border-color: rgba(0, 0, 0, 0.2); |
|||
border-style: solid; |
|||
border-width: 1px; |
|||
-webkit-border-radius: 5px; |
|||
border-radius: 5px; |
|||
-webkit-box-shadow: 0 5px 10px rgba(0, 0, 0, 0.2); |
|||
box-shadow: 0 5px 10px rgba(0, 0, 0, 0.2); |
|||
-moz-background-clip: padding; |
|||
background-clip: padding-box; |
|||
} |
|||
|
|||
.ui-menu-item > a.ui-corner-all { |
|||
display: block; |
|||
padding: 3px 15px; |
|||
clear: both; |
|||
font-weight: normal; |
|||
line-height: 18px; |
|||
color: #555555; |
|||
white-space: nowrap; |
|||
text-decoration: none; |
|||
} |
|||
|
|||
.ui-state-hover, .ui-state-active { |
|||
color: #ffffff; |
|||
text-decoration: none; |
|||
background-color: #0088cc; |
|||
border-radius: 0px; |
|||
-webkit-border-radius: 0px; |
|||
background-image: none; |
|||
} |
|||
|
|||
.logo-div { |
|||
height:67px; |
|||
width:150px; |
|||
margin-right:1%; |
|||
margin-bottom: 0%; |
|||
margin-top:1%; |
|||
margin-left:2%; |
|||
background-size: 100%; |
|||
background-size: cover; |
|||
-webkit-background-size: cover; |
|||
-o-background-size: cover; |
|||
background-size: cover; |
|||
background-position: center center; |
|||
/*background-image: url('https://mgoal.ca/goal_transp.png');*/ |
|||
} |
|||
|
|||
.title-div { |
|||
margin-right:1%; |
|||
margin-bottom: -3%; |
|||
margin-top:1%; |
|||
margin-left:30%; |
|||
font-size: 25px; |
|||
} |
|||
File diff suppressed because one or more lines are too long
@ -0,0 +1,47 @@ |
|||
{% extends "bootstrap/base.html" %} |
|||
{% block head %} |
|||
{{super()}} |
|||
<meta name="viewport" content="width=device-width, initial-scale=1"> |
|||
<header class="nav navbar"> |
|||
<section class="navbar-section"> |
|||
<div class="title-div"> |
|||
<strong>Search For Books - by the |
|||
<a href="http://localhost:8001/blog" target="_blank">Guerilla Open Access League</a> |
|||
</strong> |
|||
</div> |
|||
</section> |
|||
<section class="navbar-section"> |
|||
<search class="search-form"></search> |
|||
</section> |
|||
</header> |
|||
{% endblock %} |
|||
<html> |
|||
<body> |
|||
{% block content %} |
|||
|
|||
<results></results> |
|||
|
|||
{% endblock %} |
|||
|
|||
<footer class="footer"> |
|||
</footer> |
|||
|
|||
{% block styles %} |
|||
{{super()}} |
|||
<link rel="stylesheet" href="http://localhost:8001/styles/spectre.min.css"> |
|||
<link rel="stylesheet" href="http://localhost:8001/styles/search.css"> |
|||
{% endblock %} |
|||
|
|||
{% block scripts %} |
|||
{{super()}} |
|||
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/riot/2.4.1/riot+compiler.min.js"></script> |
|||
<script type="text/javascript" src="http://localhost:8001/jquery-ui-1.11.4.custom/jquery-ui.min.js"></script> |
|||
<script type="riot/tag" src="http://localhost:8001/scripts/search.tag"></script> |
|||
<script type="riot/tag" src="http://localhost:8001/scripts/book.tag"></script> |
|||
<script type="riot/tag" src="http://localhost:8001/scripts/class.tag"></script> |
|||
<script type="riot/tag" src="http://localhost:8001/scripts/row.tag"></script> |
|||
<script type="riot/tag" src="http://localhost:8001/scripts/results.tag"></script> |
|||
<script type="text/javascript" src="http://localhost:8001/scripts/search.js"></script> |
|||
{% endblock %} |
|||
</body> |
|||
</html> |
|||
@ -0,0 +1,24 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
class UnIndexable(Exception): |
|||
def __init__(self, course): |
|||
self.course = course |
|||
|
|||
@property |
|||
def reason(self): |
|||
course = self.course |
|||
if not course["code"] and not course["title"]: |
|||
message = "there was no course code and no title defined" |
|||
if not course["code"]: |
|||
message = "there was no course code defined" |
|||
if not course["title"]: |
|||
message = "there was no course title defined" |
|||
if not course["sections"]: |
|||
message = "there were no sections defined" |
|||
return """ |
|||
There was a problem with indexing this course. |
|||
%s |
|||
There could be several reasons why, my best guess is that %s |
|||
We need at least the course code, title, and one or more sections to index |
|||
|
|||
""" % (course, message) |
|||
@ -0,0 +1,97 @@ |
|||
#! /usr/bin/python2 |
|||
|
|||
from json import loads, load |
|||
from re import sub, split |
|||
from itertools import groupby |
|||
from numpy import mean |
|||
from operator import attrgetter |
|||
|
|||
import pygal |
|||
import csv |
|||
|
|||
class Textbook(object): |
|||
def __init__(self, dept, code, title, author, price): |
|||
self.dept = dept |
|||
self.code = code |
|||
self.title = title |
|||
self.author = author |
|||
self.price = float(price) |
|||
|
|||
def __repr__(self): |
|||
return "Dept = %s, Code = %s, %s by %s, costs $%s" % (self.dept, |
|||
self.code, |
|||
self.title, |
|||
self.author, |
|||
self.price) |
|||
|
|||
|
|||
def courses(): |
|||
with open("./books.csv", "r") as books: |
|||
booksreader = csv.reader(books) |
|||
for row in booksreader: |
|||
yield row |
|||
|
|||
|
|||
def groupDept(courselist): |
|||
sortedCourses = sorted(courselist, key=attrgetter("dept")) |
|||
for course in groupby(sortedCourses, attrgetter("dept")): |
|||
yield course[0], list(course[1]) |
|||
|
|||
def meanPrice(books): |
|||
return mean([book.price for book in books]) |
|||
|
|||
# Questions, |
|||
# mean cost per department |
|||
# mean cost per faculty |
|||
# mean difference between book store copies and other copies per dept and faculty |
|||
# number of overlapping books per faculty, do eng students benefit from that? |
|||
|
|||
# maybe a survey for students to see how often they buy books from other sources |
|||
# correlate with how much they could be saving? |
|||
|
|||
facultyDesc = { |
|||
"hum" : "Humanities", |
|||
"bus" : "Business", |
|||
"hlth" : "Health Science", |
|||
"eng" : "Engineering", |
|||
"sci" : "Science", |
|||
"socsci" : "Social Sciences", |
|||
"artsci" : "Arts & Sciences", |
|||
"meld" : "MELD" |
|||
} |
|||
|
|||
faculties = load(open("./faculties.json")) |
|||
|
|||
def categorize(dept): |
|||
# faculties |
|||
return facultyDesc.get(faculties.get(dept, False), False) |
|||
|
|||
def byFaculty(): |
|||
for dept, books in groupDept(courses()): |
|||
yield (categorize(dept), dept, books) |
|||
|
|||
def meanFacultyCosts(): |
|||
byfac = list(byFaculty()) |
|||
graph = pygal.Bar() |
|||
graph.title = "Mean textbook cost by faculty" |
|||
sortedFacs = sorted(byfac, key=lambda x: x[0]) |
|||
for fac in groupby(sortedFacs, lambda x: x[0]): |
|||
graph.add(fac[0], meanPrice(list(fac[1])[0][2])) |
|||
graph.value_formatter = lambda x: '$%.2f' % x if x is not None else "None" |
|||
return graph.render(transpose=True) |
|||
|
|||
def meanCosts(): |
|||
cs = groupDept(courses()) |
|||
graph = pygal.Bar() |
|||
graph.title = "Mean textbook cost by department" |
|||
for c in cs: |
|||
dept, books = c |
|||
graph.add(dept, meanPrice(books)) |
|||
#graph.render_to_file("./test_graph.svg") |
|||
graph.value_formatter = lambda x: '$%.2f' % x if x is not None else "None" |
|||
return graph.render_table(style=True, transpose=True) |
|||
|
|||
for x in courses(): |
|||
print x |
|||
#print meanCosts() |
|||
#print meanFacultyCosts() |
|||
@ -0,0 +1,148 @@ |
|||
#! /usr/bin/python2 |
|||
from functools import partial |
|||
from couchdb import ResourceConflict |
|||
|
|||
from flask import Flask, render_template, flash, request, send_from_directory |
|||
from flask_bootstrap import Bootstrap |
|||
from flask_appconfig import AppConfig |
|||
from urllib import unquote |
|||
from search import searchTerms |
|||
|
|||
from openlibrary import bookUrls |
|||
|
|||
from archive import searchIA |
|||
from urllib import quote, unquote |
|||
from json import dumps, loads |
|||
|
|||
from werkzeug.contrib.cache import MemcachedCache |
|||
cache = MemcachedCache(['127.0.0.1:11211']) |
|||
|
|||
import os |
|||
|
|||
def predict(fieldtype, term): |
|||
print fieldtype |
|||
print term |
|||
if not term: |
|||
return "[]" |
|||
else: |
|||
try: |
|||
cs = completers[fieldtype](term.lower()) |
|||
except KeyError: |
|||
return "[]" |
|||
if cs: |
|||
return cs |
|||
return "[]" |
|||
|
|||
def predictor(fieldtype): |
|||
def inner(request): |
|||
params = dict(request.args.items()) |
|||
return predict(fieldtype, params["term"]) |
|||
return inner |
|||
|
|||
def cacheit(key, thunk): |
|||
""" |
|||
Tries to find a cached version of ``key'' |
|||
If there is no cached version then it will |
|||
evaluate thunk (which must be a generator) |
|||
and cache that, then return the result |
|||
""" |
|||
cached = cache.get(quote(key)) |
|||
if cached is None: |
|||
result = list(thunk()) |
|||
cache.set(quote(key), result) |
|||
return result |
|||
return cached |
|||
|
|||
def ClassSearch(configfile=None): |
|||
defaults = {"Day", "Building", "Exact Location", "Department"} |
|||
app = Flask(__name__) |
|||
AppConfig(app, configfile) # Flask-Appconfig is not necessary, but |
|||
# highly recommend =) |
|||
# https://github.com/mbr/flask-appconfig |
|||
Bootstrap(app) |
|||
|
|||
app.config["scripts"] = "/home/wes/MGOAL/scripts" |
|||
app.config["styles"] = "/home/wes/MGOAL/styles" |
|||
|
|||
@app.route('/favicon.ico') |
|||
def favicon(): |
|||
return send_from_directory("/srv/http/goal/favicon.ico", |
|||
'favicon.ico', mimetype='image/vnd.microsoft.icon') |
|||
|
|||
|
|||
@app.route("/buildpred", methods=("GET", "POST")) |
|||
def buildpred(): |
|||
return predictbuild(request) |
|||
|
|||
@app.route("/locpred", methods=("GET", "POST")) |
|||
def locpred(): |
|||
return predictloc(request) |
|||
|
|||
@app.route("/daypred", methods=("GET", "POST")) |
|||
def daypred(): |
|||
return predictday(request) |
|||
|
|||
@app.route("/deptpred", methods=("GET", "POST")) |
|||
def deptpred(): |
|||
return predictdept(request) |
|||
|
|||
@app.route("/titlepred", methods=("GET", "POST")) |
|||
def titlepred(): |
|||
return predicttitle(request) |
|||
|
|||
@app.route("/", methods=("GET", "POST")) |
|||
def index(): |
|||
return render_template("search.html") |
|||
|
|||
@app.route("/fc", methods=("GET", "POST")) |
|||
def fc(): |
|||
""" Filter Courses """ |
|||
print "trying to get courses" |
|||
params = dict(request.args.items()) |
|||
for key, val in params.iteritems(): |
|||
if val in defaults: |
|||
del params[key] |
|||
results = searchTerms(params) |
|||
return results |
|||
|
|||
@app.route("/resources", methods=("GET", "POST")) |
|||
def resources(): |
|||
""" Get Resources """ |
|||
notRequired = False |
|||
params = loads(dict(request.args.items())["data"]) |
|||
print params |
|||
author = params["author"] |
|||
title = params["title"] |
|||
|
|||
if ("No Textbooks" in title or |
|||
"No Adoption" in title): |
|||
return dumps("false") |
|||
|
|||
# Cache the result of the open library search |
|||
openlib = cacheit("openlib"+title+author, lambda : bookUrls(title, author)) |
|||
print openlib |
|||
|
|||
# cache the result of an internet archive search |
|||
iarchive = cacheit("iarchive"+title+author, lambda : searchIA(title, author)) |
|||
print iarchive |
|||
|
|||
if not (any(openlib) or any(iarchive)): |
|||
# We literally could not find ANYTHING |
|||
return dumps("false") |
|||
|
|||
return dumps({ |
|||
"iarchive" : iarchive, |
|||
"openlib" : openlib |
|||
}) |
|||
|
|||
@app.route("/scripts/<filename>") |
|||
def send_script(filename): |
|||
return send_from_directory(app.config["scripts"], filename) |
|||
|
|||
@app.route("/styles/<filename>") |
|||
def send_style(filename): |
|||
return send_from_directory(app.config["styles"], filename) |
|||
return app |
|||
|
|||
if __name__ == "__main__": |
|||
ClassSearch().run(port=8001, debug=True) |
|||
Loading…
Reference in new issue